lorrystream 0.0.2__tar.gz → 0.0.3__tar.gz

lorrystream-0.0.2/CHANGES.rst → lorrystream-0.0.3/CHANGES.md

@@ -1,17 +1,15 @@
-#####################
-LorryStream changelog
-#####################
+# Changelog
 
-in progress
-===========
+## in progress
 
-2024-07-10 v0.0.2
-=================
+## 2024-08-16 v0.0.3
+- Carabas: A subsystem to divert workloads to other people’s computers
+
+## 2024-07-10 v0.0.2
 - Initial working version, supporting MQTT, AMQP, and SQLAlchemy/CrateDB
 - Add CLI interface
 - Add logging
 - Add software tests
 
-2023-04-06 v0.0.1
-=================
+## 2023-04-06 v0.0.1
 - Framework and code layout explorations

lorrystream-0.0.3/MANIFEST.in

@@ -0,0 +1,3 @@
+include CHANGES.md LICENSE README.md
+recursive-include doc *.md *.rst
+prune tests

lorrystream-0.0.3/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.1
+Name: lorrystream
+Version: 0.0.3
+Summary: A lightweight and polyglot stream-processing library, to be used as a data backplane-, message relay-, or pipeline-subsystem.
+Author-email: Andreas Motl <andreas.motl@panodata.org>, Richard Pobering <richard.pobering@panodata.org>
+License: LGPL 3, EUPL 1.2
+Project-URL: Changelog, https://lorrystream.readthedocs.io/changes.html
+Project-URL: Documentation, https://lorrystream.readthedocs.io/
+Project-URL: Homepage, https://lorrystream.readthedocs.io/
+Project-URL: Issues, https://github.com/daq-tools/lorrystream/issues
+Project-URL: Repository, https://github.com/daq-tools/lorrystream
+Keywords: amqp,data,export,import,kinesis,mqtt,pandas,rdbms,sql,stream
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Customer Service
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Manufacturing
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: System Administrators
+Classifier: Intended Audience :: Telecommunications Industry
+Classifier: License :: OSI Approved :: European Union Public Licence 1.2 (EUPL 1.2)
+Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: Unix
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Adaptive Technologies
+Classifier: Topic :: Communications
+Classifier: Topic :: Database
+Classifier: Topic :: Documentation
+Classifier: Topic :: Education
+Classifier: Topic :: Internet
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Office/Business
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Embedded Systems
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Object Brokering
+Classifier: Topic :: Software Development :: Pre-processors
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Version Control
+Classifier: Topic :: System :: Archiving
+Classifier: Topic :: System :: Benchmark
+Classifier: Topic :: System :: Clustering
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Topic :: System :: Hardware
+Classifier: Topic :: System :: Logging
+Classifier: Topic :: System :: Monitoring
+Classifier: Topic :: System :: Networking
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/x-rst
+License-File: LICENSE
+Requires-Dist: boltons
+Requires-Dist: click<9
+Requires-Dist: colorama<1
+Requires-Dist: colorlog
+Requires-Dist: commons-codec==0.0.3
+Requires-Dist: dask
+Requires-Dist: funcy
+Requires-Dist: influxdb
+Requires-Dist: influxdb-client[ciso]
+Requires-Dist: paho-mqtt
+Requires-Dist: pandas<2.3
+Requires-Dist: pika<1.4
+Requires-Dist: simplejson<4
+Requires-Dist: sqlalchemy==2.0.*
+Requires-Dist: sqlalchemy-cratedb==0.38.0
+Requires-Dist: streamz
+Requires-Dist: tomli
+Requires-Dist: toolz
+Provides-Extra: all
+Requires-Dist: lorrystream[carabas]; extra == "all"
+Provides-Extra: carabas
+Requires-Dist: aiobotocore==2.13.*; extra == "carabas"
+Requires-Dist: async-kinesis<1.2; extra == "carabas"
+Requires-Dist: aws-lambda-layer<0.6; extra == "carabas"
+Requires-Dist: boto3==1.34.*; extra == "carabas"
+Requires-Dist: cottonformation<1.2; extra == "carabas"
+Requires-Dist: localstack[base-runtime]<3.7; extra == "carabas"
+Provides-Extra: develop
+Requires-Dist: black<25; extra == "develop"
+Requires-Dist: mypy<1.12; extra == "develop"
+Requires-Dist: pika-stubs<0.2; extra == "develop"
+Requires-Dist: poethepoet<0.28; extra == "develop"
+Requires-Dist: pyproject-fmt<2.3; extra == "develop"
+Requires-Dist: ruff<0.6; extra == "develop"
+Requires-Dist: validate-pyproject<0.19; extra == "develop"
+Provides-Extra: docs
+Requires-Dist: furo==2024.1.29; extra == "docs"
+Requires-Dist: myst-parser[linkify]<5,>=0.18; extra == "docs"
+Requires-Dist: sphinx-autobuild==2021.3.14; extra == "docs"
+Requires-Dist: sphinx-copybutton; extra == "docs"
+Requires-Dist: sphinx-design-elements<1; extra == "docs"
+Requires-Dist: sphinxcontrib-mermaid<1; extra == "docs"
+Requires-Dist: sphinxext-opengraph<1; extra == "docs"
+Provides-Extra: release
+Requires-Dist: build<2; extra == "release"
+Requires-Dist: twine<6; extra == "release"
+Provides-Extra: test
+Requires-Dist: cratedb-toolkit[testing]==0.0.15; extra == "test"
+Requires-Dist: docker<7; extra == "test"
+Requires-Dist: localstack-utils<1.1; extra == "test"
+Requires-Dist: pytest<9; extra == "test"
+Requires-Dist: pytest-asyncio-cooperative<0.30; extra == "test"
+Requires-Dist: pytest-cov<6; extra == "test"
+Requires-Dist: pytest-mock<4; extra == "test"
+Requires-Dist: pytest-mqtt<0.5,>=0.4.2; extra == "test"
+Requires-Dist: requests==2.28.1; extra == "test"
+Requires-Dist: testcontainer-python-rabbitmq==0.4.*; extra == "test"

lorrystream-0.0.3/README.md

@@ -0,0 +1,247 @@
+# LorryStream
+
+## About
+
+**LorryStream** is a lightweight and polyglot stream-processing library,
+to be used as a data backplane-, message relay-, or pipeline-subsystem,
+in the spirit of [socat] and [GStreamer]. It is based on [Streamz],
+[Dask], and other Python libraries.
+
+You can use **LorryStream** to store data received from the network into
+databases, or to relay it back to the network, for example into
+different bus systems. It can be used both as a standalone program, and
+as a library.
+
+It is conceived to generalize and improve the corresponding subsystems
+of programs and frameworks like [Kotori], [Wetterdienst], [Luftdatenpumpe],
+[amqp-forward], [ttnlogger], [Kahn], or [mqttwarn].
+
+
+## Synopsis
+
+The canonical command is `lorry relay <source> <sink>`. Please note
+`%23` is `#`.
+
+``` sh
+lorry relay \
+    "mqtt://localhost/testdrive/%23" \
+    "crate://localhost/?table=testdrive"
+```
+
+If you prefer a GStreamer-like pipeline definition syntax.
+
+``` sh
+lorry launch "mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive"
+```
+
+## Quickstart
+
+If you are in a hurry, and want to run LorryStream without any
+installation, just use the OCI image on Podman or Docker.
+
+``` sh
+docker run --rm --network=host ghcr.io/daq-tools/lorrystream \
+    lorry relay \
+    "mqtt://localhost/testdrive/%23" \
+    "crate://localhost/?table=testdrive"
+```
+
+## Setup
+
+Install `lorrystream` from PyPI.
+
+``` sh
+pip install lorrystream
+```
+
+## Usage
+
+This section outlines some example invocations of LorryStream, both on
+the command line, and per library use. Other than the resources
+available from the web, testing data can be acquired from the
+repository's [testdata] folder.
+
+### Prerequisites
+
+For properly running some of the example invocations outlined below, you
+will need a few servers. The easiest way to spin up those instances is
+to use Podman or Docker.
+
+``` sh
+docker run --name=mosquitto --rm -it --publish=1883:1883 \
+    eclipse-mosquitto:2.0.15 mosquitto -c /mosquitto-no-auth.conf
+```
+
+-- <https://github.com/docker-library/docs/blob/master/eclipse-mosquitto/README.md>
+
+``` sh
+docker run --name=cratedb --rm -it --publish=4200:4200 --publish=5432:5432 \
+    crate:5.2 -Cdiscovery.type=single-node
+```
+
+-- <https://github.com/docker-library/docs/blob/master/crate/README.md>
+
+### Command line use
+
+#### Help
+
+``` sh
+lorry --help
+lorry info
+lorry relay --help
+```
+
+#### Bus to storage
+
+``` sh
+# Relay messages from MQTT to CrateDB.
+lorry relay \
+    "mqtt://localhost/testdrive/%23" \
+    "crate://localhost/?table=testdrive"
+```
+
+#### Bus to bus
+
+``` sh
+# Relay messages from AMQP to MQTT.
+lorry relay \
+    "amqp://localhost/testdrive/demo" \
+    "mqtt://localhost/testdrive/demo"
+```
+
+### Library use
+
+``` python
+>>> from lorrystream import parse_launch
+>>> parse_launch("mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive")
+```
+
+#### OCI
+
+OCI images are available on the GitHub Container Registry (GHCR). We are
+publishing image variants for general availability- and
+nightly-releases, and pull requests.
+
+In order to always run the latest `nightly` development version, and to
+use a shortcut for that, this section outlines how to use an alias for
+`lorry`, and a variable for storing the data source and sink URIs. It
+may be useful to save a few keystrokes on subsequent invocations.
+
+``` sh
+docker pull ghcr.io/daq-tools/lorrystream:nightly
+alias lorry="docker run --rm --interactive ghcr.io/daq-tools/lorrystream:nightly lorry"
+SOURCE=mqtt://localhost/testdrive/%23
+SINK=crate://crate@localhost:4200/?table=testdrive
+
+lorry relay "${SOURCE}" "${SINK}"
+```
+
+
+## Story
+
+### Details
+
+- Data sources are message bus systems like AMQP, Kafka, MQTT, ZeroMQ,
+  and network listener endpoints for TCP, UDP, HTTP, and WebSocket.
+- Data sinks are RDBMS databases supported by SQLAlchemy, or other
+  message brokers.
+
+### Motivation
+
+- Implement a reusable solution, simple to install and operate, that
+  doesn't depend on vendor-provided infrastructure, and can easily be
+  embedded into existing frameworks and software stacks, or integrated
+  otherwise by running it as a separate service.
+- Help the community and industry to modernize their aging DAQ backend
+  systems designed within the previous decades.
+- Use as pipeline elements, protocol translator, bridge elements.
+
+### Background
+
+> Flow-Based Programming ([FBP]) is a programming paradigm that uses a
+> "data processing factory" metaphor for designing and building applications.
+> It is a special case of [dataflow] programming characterized by asynchronous,
+> concurrent processes "under the covers".
+>
+> FBP has been found to support improved development time and
+> maintainability, reusability, rapid prototyping, simulation, improved
+> performance, and good communication among developers, maintenance
+> staff, users, systems people, and management - not to mention that FBP
+> naturally takes advantage of multiple cores, without the programmer
+> having to struggle with the intricacies of multitasking.
+>
+> -- [Flow-based Programming]
+
+### Caveat
+
+Please note that LorryStream is alpha-quality software, and a work in
+progress. Contributions of all kinds are very welcome, in order to make
+it more solid.
+
+Breaking changes should be expected until a 1.0 release, so version
+pinning is recommended, especially when you use it as a library.
+
+
+## Project information
+
+
+### Contributions
+
+The LorryStream library is an open source project, and is [managed on
+GitHub]. Every kind of contribution, feedback, or patch, is much welcome.
+[Create an issue] or submit a patch if you think we should include a new
+feature, or to report or fix a bug.
+
+### Development
+
+In order to setup a development environment on your workstation, please
+head over to the [development sandbox]
+documentation. When you see the software tests succeed, you should be
+ready to start hacking.
+
+### License
+
+The project is licensed under the terms of the LGPL license, see
+[LICENSE].
+
+### Prior art
+
+We are maintaining a [list of other projects] with
+the same or similar goals like LorryStream.
+
+### Kudos
+
+- [J. Paul Rodker Morrison] for discovering/inventing the Flow-Based
+  Programming ([FBP]) paradigm in the late '60s.
+
+- [Matthew Rocklin], [Christopher J. 'CJ' Wright], and [Chinmay Chandak]
+  for conceiving [Streamz].
+
+
+[amqp-forward]: https://github.com/daq-tools/amqp-forward
+[Chinmay Chandak]: https://github.com/chinmaychandak
+[Christopher J. 'CJ' Wright]: https://github.com/CJ-Wright
+[Create an issue]: https://github.com/daq-tools/lorrystream/issues
+[Dask]: https://github.com/dask/dask
+[Dataflow]: https://en.wikipedia.org/wiki/Dataflow
+[development sandbox]: https://github.com/daq-tools/lorrystream/blob/main/doc/development.rst
+[FBP]: https://en.wikipedia.org/wiki/Flow-based_programming
+[Flow-based Programming]: https://jpaulm.github.io/fbp/
+[fsspec]: https://pypi.org/project/fsspec/
+[GStreamer]: https://en.wikipedia.org/wiki/GStreamer
+[J. Paul Rodker Morrison]: https://jpaulm.github.io/
+[Kahn]: https://github.com/maritime-labs/kahn
+[Kotori]: https://github.com/daq-tools/kotori
+[LICENSE]: https://github.com/daq-tools/lorrystream/blob/main/LICENSE
+[list of other projects]: https://github.com/daq-tools/lorrystream/blob/main/doc/prior-art.rst
+[Luftdatenpumpe]: https://github.com/earthobservations/luftdatenpumpe
+[managed on GitHub]: https://github.com/daq-tools/lorrystream
+[Matthew Rocklin]: https://github.com/mrocklin
+[mqttwarn]: https://github.com/jpmens/mqttwarn
+[pandas]: https://pandas.pydata.org/
+[socat]: http://www.dest-unreach.org/socat/
+[SQLAlchemy]: https://pypi.org/project/SQLAlchemy/
+[Streamz]: https://github.com/python-streamz/streamz
+[testdata]: https://github.com/daq-tools/lorrystream/tree/main/tests/testdata
+[ttnlogger]: https://github.com/daq-tools/ttnlogger
+[Wetterdienst]: https://github.com/earthobservations/wetterdienst/

{lorrystream-0.0.2 → lorrystream-0.0.3}/doc/backlog.rst

@@ -1,6 +1,6 @@
-###################
-LorryStream backlog
-###################
+#######
+Backlog
+#######
 
 
 ***********
@@ -40,6 +40,8 @@ Iteration 2
 - [o] Examples: Add ``appsink`` example
 - [o] Improve inline docs
 - [o] Release 0.1.0
+- [o] CSV: https://github.com/alan-turing-institute/CleverCSV
+- [o] Excel & ODF: https://github.com/dimastbk/python-calamine
 
 
 ***********

lorrystream-0.0.3/doc/carabas/backlog.md

@@ -0,0 +1,21 @@
+# Carabas Backlog
+
+## Iteration +1
+- [x] Improve type mapping
+- [x] Generalize CDC event -> SQL translator
+- [ ] Only optionally display debug output of Docker build process,
+  [ ] when using `--verbose`.
+- [ ] Bring back "Zip" use, for interactive hacking
+- [ ] Distill into a Lambda variant
+- [ ] Automation!
+  - [ ] DDL: CREATE TABLE <tablename> (data OBJECT(DYNAMIC));
+  - [ ] Wrap KCL launcher into manager component
+
+## Iteration +2
+- [ ] Performance improvements (simdjson?)
+- [ ] Use SQLAlchemy for generating and submitting SQL statement
+- [ ] Improve efficiency by using bulk operations when applicable
+- [ ] is in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state and can not be updated
+- [ ] is in ROLLBACK_COMPLETE state and can not be updated.
+- [ ] Cannot create a publicly accessible DBInstance. The specified VPC has no
+  internet gateway attached.Update the VPC and then try again

lorrystream-0.0.3/doc/carabas/dms/handbook.md

@@ -0,0 +1,79 @@
+(aws-dms-handbook)=
+# AWS DMS Handbook
+
+A few useful AWSCLI commands to check the status of the DMS engine and
+relevant pipeline elements. You can also use the AWS Web Console to
+inspect and commandeer the same details.
+
+
+## Status Checks
+Display ARNs of all replication instances.
+```shell
+aws dms describe-replication-instances | jq -r '.ReplicationInstances[].ReplicationInstanceArn'
+```
+Display replication endpoints and relevant connection settings.
+```shell
+aws dms describe-endpoints
+```
+Invoke connection test on given DMS endpoint.
+```shell
+aws dms test-connection \
+  --replication-instance-arn arn:aws:dms:eu-central-1:831394476016:rep:JD2LL6OM35BJZNKZIRSOE2FXIY \
+  --endpoint-arn arn:aws:dms:eu-central-1:831394476016:endpoint:3IVDGL6E4RDNBF2LFBYF6DYV3Y
+```
+Display connection test results.
+```shell
+aws dms describe-connections
+```
+
+
+## Operations
+Enumerate all configured replication tasks with compact output.
+```shell
+aws dms describe-replication-tasks | \
+  jq '.ReplicationTasks[] | {ReplicationTaskIdentifier, ReplicationTaskArn, MigrationType, StartReplicationType, Status, StopReason, FailureMessages, ProvisionData}'
+```
+Start replication task with given ARN.
+```shell
+aws dms start-replication-task \
+  --start-replication-task-type start-replication --replication-task-arn \
+  arn:aws:dms:eu-central-1:831394476016:task:7QBLNBTPCNDEBG7CHI3WA73YFA
+```
+Stop replication task with given ARN.
+```shell
+aws dms stop-replication-task --replication-task-arn \
+  arn:aws:dms:eu-central-1:831394476016:task:7QBLNBTPCNDEBG7CHI3WA73YFA
+```
+
+
+## Logging
+To see detailed progress about the replication process, use CloudWatch to
+inspect corresponding log output.
+
+Enumerate all log groups.
+```shell
+aws logs describe-log-groups
+```
+
+Get log output history.
+```shell
+aws logs get-log-events \
+  --log-group-name dms-tasks-testdrive-dms-instance \
+  --log-stream-name dms-task-7QBLNBTPCNDEBG7CHI3WA73YFA | jq .events[].message
+```
+
+Start watching the log output using the `start-live-tail` CloudWatch operation.
+```shell
+aws logs start-live-tail --log-group-identifiers \
+    arn:aws:logs:eu-central-1:831394476016:log-group:/aws/rds/instance/testdrive-dms-postgresql-dev-db/postgresql \
+    arn:aws:logs:eu-central-1:831394476016:log-group:dms-tasks-testdrive-dms-instance
+```
+
+
+## CloudFormation
+When the CloudFormation deployment is stuck, or if you want to start from scratch,
+those commands are useful.
+```shell
+aws cloudformation continue-update-rollback --stack-name testdrive-dms-postgresql-dev
+aws cloudformation delete-stack --stack-name testdrive-dms-postgresql-dev
+```

lorrystream-0.0.3/doc/carabas/dms/index.md

@@ -0,0 +1,135 @@
+(aws-dms)=
+# Pipelines with AWS DMS
+
+_AWS DMS to Kinesis to CrateDB._
+
+## What's Inside
+- [Working with AWS DMS tasks]
+- [Using Amazon Kinesis Data Streams as a target for AWS Database Migration Service]
+- An IaC driver program based on [AWS CloudFormation] technologies using the
+  [cottonformation] Python API. It can be used to set up infrastructure on AWS
+  without much ado.
+- DMS: Full load and CDC
+- DMS Source: RDS PostgreSQL
+- DMS Target: Amazon Kinesis
+- CDC Target: CrateDB Cloud
+
+
+## AWS Infrastructure Setup
+The following walkthrough describes a full deployment of AWS DMS including
+relevant outbound data processors for demonstration purposes.
+
+In order to run it in production, you are welcome to derive from it and tweak
+it for your own purposes. YMMV. If you need support, don't hesitate to ask for
+help.
+
+### Install
+Install LorryStream.
+```shell
+pip install --upgrade 'lorrystream[carabas]'
+```
+Acquire IaC driver program.
+```shell
+wget https://github.com/daq-tools/lorrystream/raw/main/examples/aws/rds_postgresql_kinesis_lambda_oci_cratedb.py
+```
+
+### Configure
+Please configure endpoint and replication settings within the source code
+of the IaC program you just acquired, and presented next.
+
+### Deploy
+First, prepare an AWS ECR repository for publishing the OCI image including your
+downstream processor element that is consuming the replication data stream from
+Amazon Kinesis, and runs it into CrateDB. To learn about how this works, please
+visit the documentation section about the [](project:#ecr-repository).
+
+Configure CrateDB database sink address.
+```shell
+export SINK_SQLALCHEMY_URL='crate://admin:dZ..qB@example.eks1.eu-west-1.aws.cratedb.net:4200/?ssl=true'
+```
+
+Invoke the IaC driver program in order to deploy relevant resources on AWS.
+```shell
+python examples/aws/rds_postgresql_kinesis_lambda_oci_cratedb.py
+```
+
+After deployment succeeded, you will be presented a corresponding response including
+relevant information about entrypoints to the software stack you've just created.
+```text
+Result of CloudFormation deployment:
+psql command: psql "postgresql://dynapipe:secret11@testdrive-dms-postgresql-dev-db.czylftvqn1ed.eu-central-1.rds.amazonaws.com:5432/postgres"
+RDS Instance ARN: arn:aws:rds:eu-central-1:831394476016:db:testdrive-dms-postgresql-dev-db
+Stream ARN: arn:aws:kinesis:eu-central-1:831394476016:stream/testdrive-dms-postgresql-dev-stream
+Replication ARN: arn:aws:dms:eu-central-1:831394476016:replication-config:EAM3JEHXGBGZBPN5PLON7NPDEE
+```
+
+:::{note}
+Please note this is a demonstration stack, deviating from typical real-world situations.
+
+- Contrary to this stack, which includes an RDS PostgreSQL instance, a database instance
+  will already be up and running, so the remaining task is to just configure the Kinesis
+  Data Stream and consume it.
+
+- Contrary to this stack, which uses AWS Lambda to host the downstream processor element,
+  when aiming for better cost-effectiveness, you will run corresponding code on a dedicated
+  computing environment.
+:::
+
+
+## Operations
+Please consult the [](project:#aws-dms-handbook) to learn about commands
+suitable for operating the AWS DMS engine.
+
+:::{toctree}
+:hidden:
+
+handbook
+:::
+
+
+
+## Usage
+
+### DMS
+AWS DMS provides `full-load` and `full-load-and-cdc` migration types.
+For a `full-load-and-cdc` task, AWS DMS migrates table data, and then applies
+data changes that occur on the source, automatically establishing continuous
+replication.
+
+When starting a replication task using [StartReplicationTask], you can use those
+possible values for `--start-replication-task-type`, see also [start-replication-task]:
+
+:start-replication:
+    The only valid value for the first run of the task when the migration type is
+    `full-load` or `full-load-and-cdc`
+
+:resume-processing:
+    Not applicable for any full-load task, because you can't resume partially loaded
+    tables during the full load phase. Use it to replicate the changes from the last
+    stop position.
+    
+:reload-target:
+    For a `full-load-and-cdc` task, load all the tables again, and start capturing
+    source changes.
+
+
+## Migration by DMS Source
+This section enumerates specific information to consider when aiming to use DMS
+for your database as a source element.
+
+:::{toctree}
+:maxdepth: 2
+
+postgresql
+mysql
+:::
+
+
+
+[AWS CloudFormation]: https://en.wikipedia.org/wiki/AWS_CloudFormation
+[cottonformation]: https://pypi.org/project/cottonformation/
+[StartReplicationTask]: https://docs.aws.amazon.com/dms/latest/APIReference/API_StartReplicationTask.html
+[start-replication-task]: https://docs.aws.amazon.com/cli/latest/reference/dms/start-replication-task.html
+[Using Amazon Kinesis Data Streams as a target for AWS Database Migration Service]: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html
+[Using object mapping to migrate data to a Kinesis data stream]: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html#CHAP_Target.Kinesis.ObjectMapping
+[Working with AWS DMS tasks]: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.html

lorrystream-0.0.3/doc/carabas/dms/mysql.md

@@ -0,0 +1,4 @@
+(aws-dms-mysql)=
+# AWS DMS with MySQL/MariaDB source
+
+WIP.