airbyte-source-gitlab 2.1.1__tar.gz → 2.1.2__tar.gz

Sign up to get free protection for your applications and to get access to all the features.
Files changed (51) hide show
  1. airbyte_source_gitlab-2.1.2/PKG-INFO +111 -0
  2. airbyte_source_gitlab-2.1.2/README.md +91 -0
  3. airbyte_source_gitlab-2.1.2/pyproject.toml +34 -0
  4. airbyte-source-gitlab-2.1.1/PKG-INFO +0 -141
  5. airbyte-source-gitlab-2.1.1/README.md +0 -147
  6. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/PKG-INFO +0 -141
  7. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/SOURCES.txt +0 -47
  8. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/dependency_links.txt +0 -1
  9. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/entry_points.txt +0 -2
  10. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/requires.txt +0 -8
  11. airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/top_level.txt +0 -2
  12. airbyte-source-gitlab-2.1.1/integration_tests/__init__.py +0 -0
  13. airbyte-source-gitlab-2.1.1/integration_tests/abnormal_state.json +0 -54
  14. airbyte-source-gitlab-2.1.1/integration_tests/acceptance.py +0 -16
  15. airbyte-source-gitlab-2.1.1/integration_tests/configured_catalog.json +0 -271
  16. airbyte-source-gitlab-2.1.1/integration_tests/incremental_catalog.json +0 -60
  17. airbyte-source-gitlab-2.1.1/integration_tests/invalid_config.json +0 -11
  18. airbyte-source-gitlab-2.1.1/integration_tests/sample_config.json +0 -7
  19. airbyte-source-gitlab-2.1.1/integration_tests/sample_state.json +0 -22
  20. airbyte-source-gitlab-2.1.1/setup.cfg +0 -137
  21. airbyte-source-gitlab-2.1.1/setup.py +0 -37
  22. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/__init__.py +0 -0
  23. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/config_migrations.py +0 -0
  24. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/run.py +0 -0
  25. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/branches.json +0 -0
  26. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/commits.json +0 -0
  27. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/deployments.json +0 -0
  28. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/epic_issues.json +0 -0
  29. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/epics.json +0 -0
  30. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/group_issue_boards.json +0 -0
  31. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/group_labels.json +0 -0
  32. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/group_members.json +0 -0
  33. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/group_milestones.json +0 -0
  34. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/groups.json +0 -0
  35. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/issues.json +0 -0
  36. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/jobs.json +0 -0
  37. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/merge_request_commits.json +0 -0
  38. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/merge_requests.json +0 -0
  39. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/pipelines.json +0 -0
  40. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/pipelines_extended.json +0 -0
  41. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/project_labels.json +0 -0
  42. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/project_members.json +0 -0
  43. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/project_milestones.json +0 -0
  44. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/projects.json +0 -0
  45. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/releases.json +0 -0
  46. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/tags.json +0 -0
  47. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/schemas/users.json +0 -0
  48. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/source.py +0 -0
  49. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/spec.json +0 -0
  50. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/streams.py +0 -0
  51. {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-2.1.2}/source_gitlab/utils.py +0 -0
@@ -0,0 +1,111 @@
1
+ Metadata-Version: 2.1
2
+ Name: airbyte-source-gitlab
3
+ Version: 2.1.2
4
+ Summary: Source implementation for Gitlab.
5
+ Home-page: https://airbyte.com
6
+ License: MIT
7
+ Author: Airbyte
8
+ Author-email: contact@airbyte.io
9
+ Requires-Python: >=3.9,<3.12
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Programming Language :: Python :: 3.9
13
+ Classifier: Programming Language :: Python :: 3.10
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Requires-Dist: airbyte-cdk (==0.58.8)
16
+ Requires-Dist: vcrpy (==4.1.1)
17
+ Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/gitlab
18
+ Project-URL: Repository, https://github.com/airbytehq/airbyte
19
+ Description-Content-Type: text/markdown
20
+
21
+ # Gitlab source connector
22
+
23
+
24
+ This is the repository for the Gitlab source connector, written in Python.
25
+ For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/gitlab).
26
+
27
+ ## Local development
28
+
29
+ ### Prerequisites
30
+ * Python (~=3.9)
31
+ * Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
32
+
33
+
34
+ ### Installing the connector
35
+ From this connector directory, run:
36
+ ```bash
37
+ poetry install --with dev
38
+ ```
39
+
40
+
41
+ ### Create credentials
42
+ **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/gitlab)
43
+ to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.yaml` file.
44
+ Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
45
+ See `sample_files/sample_config.json` for a sample config file.
46
+
47
+
48
+ ### Locally running the connector
49
+ ```
50
+ poetry run source-gitlab spec
51
+ poetry run source-gitlab check --config secrets/config.json
52
+ poetry run source-gitlab discover --config secrets/config.json
53
+ poetry run source-gitlab read --config secrets/config.json --catalog sample_files/configured_catalog.json
54
+ ```
55
+
56
+ ### Running unit tests
57
+ To run unit tests locally, from the connector directory run:
58
+ ```
59
+ poetry run pytest unit_tests
60
+ ```
61
+
62
+ ### Building the docker image
63
+ 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
64
+ 2. Run the following command to build the docker image:
65
+ ```bash
66
+ airbyte-ci connectors --name=source-gitlab build
67
+ ```
68
+
69
+ An image will be available on your host with the tag `airbyte/source-gitlab:dev`.
70
+
71
+
72
+ ### Running as a docker container
73
+ Then run any of the connector commands as follows:
74
+ ```
75
+ docker run --rm airbyte/source-gitlab:dev spec
76
+ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
77
+ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
78
+ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
79
+ ```
80
+
81
+ ### Running our CI test suite
82
+ You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
83
+ ```bash
84
+ airbyte-ci connectors --name=source-gitlab test
85
+ ```
86
+
87
+ ### Customizing acceptance Tests
88
+ Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
89
+ If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
90
+
91
+ ### Dependency Management
92
+ All of your dependencies should be managed via Poetry.
93
+ To add a new dependency, run:
94
+ ```bash
95
+ poetry add <package-name>
96
+ ```
97
+
98
+ Please commit the changes to `pyproject.toml` and `poetry.lock` files.
99
+
100
+ ## Publishing a new version of the connector
101
+ You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
102
+ 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
103
+ 2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
104
+ - bump the `dockerImageTag` value in in `metadata.yaml`
105
+ - bump the `version` value in `pyproject.toml`
106
+ 3. Make sure the `metadata.yaml` content is up to date.
107
+ 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
108
+ 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
109
+ 6. Pat yourself on the back for being an awesome contributor.
110
+ 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
111
+ 8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
@@ -0,0 +1,91 @@
1
+ # Gitlab source connector
2
+
3
+
4
+ This is the repository for the Gitlab source connector, written in Python.
5
+ For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/gitlab).
6
+
7
+ ## Local development
8
+
9
+ ### Prerequisites
10
+ * Python (~=3.9)
11
+ * Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
12
+
13
+
14
+ ### Installing the connector
15
+ From this connector directory, run:
16
+ ```bash
17
+ poetry install --with dev
18
+ ```
19
+
20
+
21
+ ### Create credentials
22
+ **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/gitlab)
23
+ to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.yaml` file.
24
+ Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
25
+ See `sample_files/sample_config.json` for a sample config file.
26
+
27
+
28
+ ### Locally running the connector
29
+ ```
30
+ poetry run source-gitlab spec
31
+ poetry run source-gitlab check --config secrets/config.json
32
+ poetry run source-gitlab discover --config secrets/config.json
33
+ poetry run source-gitlab read --config secrets/config.json --catalog sample_files/configured_catalog.json
34
+ ```
35
+
36
+ ### Running unit tests
37
+ To run unit tests locally, from the connector directory run:
38
+ ```
39
+ poetry run pytest unit_tests
40
+ ```
41
+
42
+ ### Building the docker image
43
+ 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
44
+ 2. Run the following command to build the docker image:
45
+ ```bash
46
+ airbyte-ci connectors --name=source-gitlab build
47
+ ```
48
+
49
+ An image will be available on your host with the tag `airbyte/source-gitlab:dev`.
50
+
51
+
52
+ ### Running as a docker container
53
+ Then run any of the connector commands as follows:
54
+ ```
55
+ docker run --rm airbyte/source-gitlab:dev spec
56
+ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
57
+ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
58
+ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
59
+ ```
60
+
61
+ ### Running our CI test suite
62
+ You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
63
+ ```bash
64
+ airbyte-ci connectors --name=source-gitlab test
65
+ ```
66
+
67
+ ### Customizing acceptance Tests
68
+ Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
69
+ If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
70
+
71
+ ### Dependency Management
72
+ All of your dependencies should be managed via Poetry.
73
+ To add a new dependency, run:
74
+ ```bash
75
+ poetry add <package-name>
76
+ ```
77
+
78
+ Please commit the changes to `pyproject.toml` and `poetry.lock` files.
79
+
80
+ ## Publishing a new version of the connector
81
+ You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
82
+ 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
83
+ 2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
84
+ - bump the `dockerImageTag` value in in `metadata.yaml`
85
+ - bump the `version` value in `pyproject.toml`
86
+ 3. Make sure the `metadata.yaml` content is up to date.
87
+ 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
88
+ 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
89
+ 6. Pat yourself on the back for being an awesome contributor.
90
+ 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
91
+ 8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
@@ -0,0 +1,34 @@
1
+ [build-system]
2
+ requires = [
3
+ "poetry-core>=1.0.0",
4
+ ]
5
+ build-backend = "poetry.core.masonry.api"
6
+
7
+ [tool.poetry]
8
+ version = "2.1.2"
9
+ name = "airbyte-source-gitlab"
10
+ description = "Source implementation for Gitlab."
11
+ authors = [
12
+ "Airbyte <contact@airbyte.io>",
13
+ ]
14
+ license = "MIT"
15
+ readme = "README.md"
16
+ documentation = "https://docs.airbyte.com/integrations/sources/gitlab"
17
+ homepage = "https://airbyte.com"
18
+ repository = "https://github.com/airbytehq/airbyte"
19
+ packages = [
20
+ { include = "source_gitlab" },
21
+ ]
22
+
23
+ [tool.poetry.dependencies]
24
+ python = "^3.9,<3.12"
25
+ airbyte-cdk = "==0.58.8"
26
+ vcrpy = "==4.1.1"
27
+
28
+ [tool.poetry.scripts]
29
+ source-gitlab = "source_gitlab.run:run"
30
+
31
+ [tool.poetry.group.dev.dependencies]
32
+ pytest-mock = "^3.12.0"
33
+ requests-mock = "^1.9.3"
34
+ pytest = "^6.1"
@@ -1,141 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: airbyte-source-gitlab
3
- Version: 2.1.1
4
- Summary: Source implementation for Gitlab.
5
- Author: Airbyte
6
- Author-email: contact@airbyte.io
7
- Description-Content-Type: text/markdown
8
- Requires-Dist: airbyte-cdk
9
- Requires-Dist: vcrpy==4.1.1
10
- Provides-Extra: tests
11
- Requires-Dist: requests-mock~=1.9.3; extra == "tests"
12
- Requires-Dist: pytest~=6.1; extra == "tests"
13
- Requires-Dist: requests_mock; extra == "tests"
14
- Requires-Dist: pytest-mock; extra == "tests"
15
-
16
- # Gitlab Source
17
-
18
- This is the repository for the Gitlab source connector, written in Python.
19
- For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/gitlab).
20
-
21
-
22
- **To iterate on this connector, make sure to complete this prerequisites section.**
23
-
24
- From this connector directory, create a virtual environment:
25
- ```
26
- python -m venv .venv
27
- ```
28
-
29
- This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
30
- development environment of choice. To activate it from the terminal, run:
31
- ```
32
- source .venv/bin/activate
33
- pip install -r requirements.txt
34
- ```
35
- If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
36
-
37
- Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
38
- used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
39
- If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
40
- should work as you expect.
41
-
42
- **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/gitlab)
43
- to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.json` file.
44
- Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
45
- See `integration_tests/sample_config.json` for a sample config file.
46
-
47
- **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source gitlab test creds`
48
- and place them into `secrets/config.json`.
49
-
50
- ```
51
- python main.py spec
52
- python main.py check --config secrets/config.json
53
- python main.py discover --config secrets/config.json
54
- python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
55
- ```
56
-
57
-
58
-
59
-
60
- The Airbyte way of building this connector is to use our `airbyte-ci` tool.
61
- You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
62
- Then running the following command will build your connector:
63
-
64
- ```bash
65
- airbyte-ci connectors --name=source-gitlab build
66
- ```
67
- Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-gitlab:dev`.
68
-
69
- When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
70
- You can customize our build process by adding a `build_customization.py` module to your connector.
71
- This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
72
- It will be imported at runtime by our build process and the functions will be called if they exist.
73
-
74
- Here is an example of a `build_customization.py` module:
75
- ```python
76
- from __future__ import annotations
77
-
78
- from typing import TYPE_CHECKING
79
-
80
- if TYPE_CHECKING:
81
- from dagger import Container
82
-
83
-
84
- async def pre_connector_install(base_image_container: Container) -> Container:
85
- return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
86
-
87
- async def post_connector_install(connector_container: Container) -> Container:
88
- return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
89
- ```
90
-
91
- This connector is built using our dynamic built process in `airbyte-ci`.
92
- The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
93
- The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
94
- It does not rely on a Dockerfile.
95
-
96
- If you would like to patch our connector and build your own a simple approach would be to:
97
-
98
- 1. Create your own Dockerfile based on the latest version of the connector image.
99
- ```Dockerfile
100
- FROM airbyte/source-gitlab:latest
101
-
102
- COPY . ./airbyte/integration_code
103
- RUN pip install ./airbyte/integration_code
104
-
105
- ```
106
- Please use this as an example. This is not optimized.
107
-
108
- 2. Build your image:
109
- ```bash
110
- docker build -t airbyte/source-gitlab:dev .
111
- docker run airbyte/source-gitlab:dev spec
112
- ```
113
- Then run any of the connector commands as follows:
114
- ```
115
- docker run --rm airbyte/source-gitlab:dev spec
116
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
117
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
118
- docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
119
- ```
120
-
121
- You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
122
- ```bash
123
- airbyte-ci connectors --name=source-gitlab test
124
- ```
125
-
126
- Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
127
- If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
128
-
129
- All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
130
- We split dependencies between two groups, dependencies that are:
131
- * required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
132
- * required for the testing need to go to `TEST_REQUIREMENTS` list
133
-
134
- You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
135
- 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
136
- 2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
137
- 3. Make sure the `metadata.yaml` content is up to date.
138
- 4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
139
- 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
140
- 6. Pat yourself on the back for being an awesome contributor.
141
- 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
@@ -1,147 +0,0 @@
1
- # Gitlab Source
2
-
3
- This is the repository for the Gitlab source connector, written in Python.
4
- For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/gitlab).
5
-
6
- ## Local development
7
-
8
- ### Prerequisites
9
- **To iterate on this connector, make sure to complete this prerequisites section.**
10
-
11
- #### Build & Activate Virtual Environment and install dependencies
12
- From this connector directory, create a virtual environment:
13
- ```
14
- python -m venv .venv
15
- ```
16
-
17
- This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
18
- development environment of choice. To activate it from the terminal, run:
19
- ```
20
- source .venv/bin/activate
21
- pip install -r requirements.txt
22
- ```
23
- If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
24
-
25
- Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
26
- used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
27
- If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
28
- should work as you expect.
29
-
30
- #### Create credentials
31
- **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/gitlab)
32
- to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.json` file.
33
- Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
34
- See `integration_tests/sample_config.json` for a sample config file.
35
-
36
- **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source gitlab test creds`
37
- and place them into `secrets/config.json`.
38
-
39
- ### Locally running the connector
40
- ```
41
- python main.py spec
42
- python main.py check --config secrets/config.json
43
- python main.py discover --config secrets/config.json
44
- python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
45
- ```
46
-
47
- ### Locally running the connector docker image
48
-
49
-
50
-
51
- #### Use `airbyte-ci` to build your connector
52
- The Airbyte way of building this connector is to use our `airbyte-ci` tool.
53
- You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
54
- Then running the following command will build your connector:
55
-
56
- ```bash
57
- airbyte-ci connectors --name=source-gitlab build
58
- ```
59
- Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-gitlab:dev`.
60
-
61
- ##### Customizing our build process
62
- When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
63
- You can customize our build process by adding a `build_customization.py` module to your connector.
64
- This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
65
- It will be imported at runtime by our build process and the functions will be called if they exist.
66
-
67
- Here is an example of a `build_customization.py` module:
68
- ```python
69
- from __future__ import annotations
70
-
71
- from typing import TYPE_CHECKING
72
-
73
- if TYPE_CHECKING:
74
- # Feel free to check the dagger documentation for more information on the Container object and its methods.
75
- # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
76
- from dagger import Container
77
-
78
-
79
- async def pre_connector_install(base_image_container: Container) -> Container:
80
- return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
81
-
82
- async def post_connector_install(connector_container: Container) -> Container:
83
- return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
84
- ```
85
-
86
- #### Build your own connector image
87
- This connector is built using our dynamic built process in `airbyte-ci`.
88
- The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
89
- The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
90
- It does not rely on a Dockerfile.
91
-
92
- If you would like to patch our connector and build your own a simple approach would be to:
93
-
94
- 1. Create your own Dockerfile based on the latest version of the connector image.
95
- ```Dockerfile
96
- FROM airbyte/source-gitlab:latest
97
-
98
- COPY . ./airbyte/integration_code
99
- RUN pip install ./airbyte/integration_code
100
-
101
- # The entrypoint and default env vars are already set in the base image
102
- # ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
103
- # ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
104
- ```
105
- Please use this as an example. This is not optimized.
106
-
107
- 2. Build your image:
108
- ```bash
109
- docker build -t airbyte/source-gitlab:dev .
110
- # Running the spec command against your patched connector
111
- docker run airbyte/source-gitlab:dev spec
112
- ```
113
- #### Run
114
- Then run any of the connector commands as follows:
115
- ```
116
- docker run --rm airbyte/source-gitlab:dev spec
117
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
118
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
119
- docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
120
- ```
121
-
122
- ## Testing
123
- You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
124
- ```bash
125
- airbyte-ci connectors --name=source-gitlab test
126
- ```
127
-
128
- ### Customizing acceptance Tests
129
- Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
130
- If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
131
-
132
- ## Dependency Management
133
- All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
134
- We split dependencies between two groups, dependencies that are:
135
- * required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
136
- * required for the testing need to go to `TEST_REQUIREMENTS` list
137
-
138
- ### Publishing a new version of the connector
139
- You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
140
- 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
141
- 2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
142
- 3. Make sure the `metadata.yaml` content is up to date.
143
- 4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
144
- 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
145
- 6. Pat yourself on the back for being an awesome contributor.
146
- 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
147
-
@@ -1,141 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: airbyte-source-gitlab
3
- Version: 2.1.1
4
- Summary: Source implementation for Gitlab.
5
- Author: Airbyte
6
- Author-email: contact@airbyte.io
7
- Description-Content-Type: text/markdown
8
- Requires-Dist: airbyte-cdk
9
- Requires-Dist: vcrpy==4.1.1
10
- Provides-Extra: tests
11
- Requires-Dist: requests-mock~=1.9.3; extra == "tests"
12
- Requires-Dist: pytest~=6.1; extra == "tests"
13
- Requires-Dist: requests_mock; extra == "tests"
14
- Requires-Dist: pytest-mock; extra == "tests"
15
-
16
- # Gitlab Source
17
-
18
- This is the repository for the Gitlab source connector, written in Python.
19
- For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/gitlab).
20
-
21
-
22
- **To iterate on this connector, make sure to complete this prerequisites section.**
23
-
24
- From this connector directory, create a virtual environment:
25
- ```
26
- python -m venv .venv
27
- ```
28
-
29
- This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
30
- development environment of choice. To activate it from the terminal, run:
31
- ```
32
- source .venv/bin/activate
33
- pip install -r requirements.txt
34
- ```
35
- If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
36
-
37
- Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
38
- used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
39
- If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
40
- should work as you expect.
41
-
42
- **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/gitlab)
43
- to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.json` file.
44
- Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
45
- See `integration_tests/sample_config.json` for a sample config file.
46
-
47
- **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source gitlab test creds`
48
- and place them into `secrets/config.json`.
49
-
50
- ```
51
- python main.py spec
52
- python main.py check --config secrets/config.json
53
- python main.py discover --config secrets/config.json
54
- python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
55
- ```
56
-
57
-
58
-
59
-
60
- The Airbyte way of building this connector is to use our `airbyte-ci` tool.
61
- You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
62
- Then running the following command will build your connector:
63
-
64
- ```bash
65
- airbyte-ci connectors --name=source-gitlab build
66
- ```
67
- Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-gitlab:dev`.
68
-
69
- When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
70
- You can customize our build process by adding a `build_customization.py` module to your connector.
71
- This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
72
- It will be imported at runtime by our build process and the functions will be called if they exist.
73
-
74
- Here is an example of a `build_customization.py` module:
75
- ```python
76
- from __future__ import annotations
77
-
78
- from typing import TYPE_CHECKING
79
-
80
- if TYPE_CHECKING:
81
- from dagger import Container
82
-
83
-
84
- async def pre_connector_install(base_image_container: Container) -> Container:
85
- return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
86
-
87
- async def post_connector_install(connector_container: Container) -> Container:
88
- return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
89
- ```
90
-
91
- This connector is built using our dynamic built process in `airbyte-ci`.
92
- The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
93
- The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
94
- It does not rely on a Dockerfile.
95
-
96
- If you would like to patch our connector and build your own a simple approach would be to:
97
-
98
- 1. Create your own Dockerfile based on the latest version of the connector image.
99
- ```Dockerfile
100
- FROM airbyte/source-gitlab:latest
101
-
102
- COPY . ./airbyte/integration_code
103
- RUN pip install ./airbyte/integration_code
104
-
105
- ```
106
- Please use this as an example. This is not optimized.
107
-
108
- 2. Build your image:
109
- ```bash
110
- docker build -t airbyte/source-gitlab:dev .
111
- docker run airbyte/source-gitlab:dev spec
112
- ```
113
- Then run any of the connector commands as follows:
114
- ```
115
- docker run --rm airbyte/source-gitlab:dev spec
116
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
117
- docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
118
- docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
119
- ```
120
-
121
- You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
122
- ```bash
123
- airbyte-ci connectors --name=source-gitlab test
124
- ```
125
-
126
- Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
127
- If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
128
-
129
- All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
130
- We split dependencies between two groups, dependencies that are:
131
- * required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
132
- * required for the testing need to go to `TEST_REQUIREMENTS` list
133
-
134
- You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
135
- 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
136
- 2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
137
- 3. Make sure the `metadata.yaml` content is up to date.
138
- 4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
139
- 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
140
- 6. Pat yourself on the back for being an awesome contributor.
141
- 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.