airbyte-source-gitlab 2.1.1__tar.gz → 3.0.0__tar.gz
Sign up to get free protection for your applications and to get access to all the features.
- airbyte_source_gitlab-3.0.0/PKG-INFO +111 -0
- airbyte_source_gitlab-3.0.0/README.md +91 -0
- airbyte_source_gitlab-3.0.0/pyproject.toml +34 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/config_migrations.py +0 -2
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/epic_issues.json +6 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/groups.json +20 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/issues.json +30 -0
- airbyte_source_gitlab-3.0.0/source_gitlab/schemas/merge_request_commits.json +85 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/merge_requests.json +3 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/pipelines_extended.json +3 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/projects.json +18 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/streams.py +3 -1
- airbyte-source-gitlab-2.1.1/PKG-INFO +0 -141
- airbyte-source-gitlab-2.1.1/README.md +0 -147
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/PKG-INFO +0 -141
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/SOURCES.txt +0 -47
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/dependency_links.txt +0 -1
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/entry_points.txt +0 -2
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/requires.txt +0 -8
- airbyte-source-gitlab-2.1.1/airbyte_source_gitlab.egg-info/top_level.txt +0 -2
- airbyte-source-gitlab-2.1.1/integration_tests/__init__.py +0 -0
- airbyte-source-gitlab-2.1.1/integration_tests/abnormal_state.json +0 -54
- airbyte-source-gitlab-2.1.1/integration_tests/acceptance.py +0 -16
- airbyte-source-gitlab-2.1.1/integration_tests/configured_catalog.json +0 -271
- airbyte-source-gitlab-2.1.1/integration_tests/incremental_catalog.json +0 -60
- airbyte-source-gitlab-2.1.1/integration_tests/invalid_config.json +0 -11
- airbyte-source-gitlab-2.1.1/integration_tests/sample_config.json +0 -7
- airbyte-source-gitlab-2.1.1/integration_tests/sample_state.json +0 -22
- airbyte-source-gitlab-2.1.1/setup.cfg +0 -137
- airbyte-source-gitlab-2.1.1/setup.py +0 -37
- airbyte-source-gitlab-2.1.1/source_gitlab/schemas/merge_request_commits.json +0 -234
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/__init__.py +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/run.py +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/branches.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/commits.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/deployments.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/epics.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/group_issue_boards.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/group_labels.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/group_members.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/group_milestones.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/jobs.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/pipelines.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/project_labels.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/project_members.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/project_milestones.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/releases.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/tags.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/users.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/source.py +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/spec.json +0 -0
- {airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/utils.py +0 -0
@@ -0,0 +1,111 @@
|
|
1
|
+
Metadata-Version: 2.1
|
2
|
+
Name: airbyte-source-gitlab
|
3
|
+
Version: 3.0.0
|
4
|
+
Summary: Source implementation for Gitlab.
|
5
|
+
Home-page: https://airbyte.com
|
6
|
+
License: MIT
|
7
|
+
Author: Airbyte
|
8
|
+
Author-email: contact@airbyte.io
|
9
|
+
Requires-Python: >=3.9,<3.12
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
12
|
+
Classifier: Programming Language :: Python :: 3.9
|
13
|
+
Classifier: Programming Language :: Python :: 3.10
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
15
|
+
Requires-Dist: airbyte-cdk (==0.58.8)
|
16
|
+
Requires-Dist: vcrpy (==4.1.1)
|
17
|
+
Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/gitlab
|
18
|
+
Project-URL: Repository, https://github.com/airbytehq/airbyte
|
19
|
+
Description-Content-Type: text/markdown
|
20
|
+
|
21
|
+
# Gitlab source connector
|
22
|
+
|
23
|
+
|
24
|
+
This is the repository for the Gitlab source connector, written in Python.
|
25
|
+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/gitlab).
|
26
|
+
|
27
|
+
## Local development
|
28
|
+
|
29
|
+
### Prerequisites
|
30
|
+
* Python (~=3.9)
|
31
|
+
* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
|
32
|
+
|
33
|
+
|
34
|
+
### Installing the connector
|
35
|
+
From this connector directory, run:
|
36
|
+
```bash
|
37
|
+
poetry install --with dev
|
38
|
+
```
|
39
|
+
|
40
|
+
|
41
|
+
### Create credentials
|
42
|
+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/gitlab)
|
43
|
+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.yaml` file.
|
44
|
+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
|
45
|
+
See `sample_files/sample_config.json` for a sample config file.
|
46
|
+
|
47
|
+
|
48
|
+
### Locally running the connector
|
49
|
+
```
|
50
|
+
poetry run source-gitlab spec
|
51
|
+
poetry run source-gitlab check --config secrets/config.json
|
52
|
+
poetry run source-gitlab discover --config secrets/config.json
|
53
|
+
poetry run source-gitlab read --config secrets/config.json --catalog sample_files/configured_catalog.json
|
54
|
+
```
|
55
|
+
|
56
|
+
### Running unit tests
|
57
|
+
To run unit tests locally, from the connector directory run:
|
58
|
+
```
|
59
|
+
poetry run pytest unit_tests
|
60
|
+
```
|
61
|
+
|
62
|
+
### Building the docker image
|
63
|
+
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
|
64
|
+
2. Run the following command to build the docker image:
|
65
|
+
```bash
|
66
|
+
airbyte-ci connectors --name=source-gitlab build
|
67
|
+
```
|
68
|
+
|
69
|
+
An image will be available on your host with the tag `airbyte/source-gitlab:dev`.
|
70
|
+
|
71
|
+
|
72
|
+
### Running as a docker container
|
73
|
+
Then run any of the connector commands as follows:
|
74
|
+
```
|
75
|
+
docker run --rm airbyte/source-gitlab:dev spec
|
76
|
+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
|
77
|
+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
|
78
|
+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
|
79
|
+
```
|
80
|
+
|
81
|
+
### Running our CI test suite
|
82
|
+
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
|
83
|
+
```bash
|
84
|
+
airbyte-ci connectors --name=source-gitlab test
|
85
|
+
```
|
86
|
+
|
87
|
+
### Customizing acceptance Tests
|
88
|
+
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
|
89
|
+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
|
90
|
+
|
91
|
+
### Dependency Management
|
92
|
+
All of your dependencies should be managed via Poetry.
|
93
|
+
To add a new dependency, run:
|
94
|
+
```bash
|
95
|
+
poetry add <package-name>
|
96
|
+
```
|
97
|
+
|
98
|
+
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
|
99
|
+
|
100
|
+
## Publishing a new version of the connector
|
101
|
+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
|
102
|
+
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
|
103
|
+
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
|
104
|
+
- bump the `dockerImageTag` value in in `metadata.yaml`
|
105
|
+
- bump the `version` value in `pyproject.toml`
|
106
|
+
3. Make sure the `metadata.yaml` content is up to date.
|
107
|
+
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
|
108
|
+
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
|
109
|
+
6. Pat yourself on the back for being an awesome contributor.
|
110
|
+
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|
111
|
+
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
|
@@ -0,0 +1,91 @@
|
|
1
|
+
# Gitlab source connector
|
2
|
+
|
3
|
+
|
4
|
+
This is the repository for the Gitlab source connector, written in Python.
|
5
|
+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/gitlab).
|
6
|
+
|
7
|
+
## Local development
|
8
|
+
|
9
|
+
### Prerequisites
|
10
|
+
* Python (~=3.9)
|
11
|
+
* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
|
12
|
+
|
13
|
+
|
14
|
+
### Installing the connector
|
15
|
+
From this connector directory, run:
|
16
|
+
```bash
|
17
|
+
poetry install --with dev
|
18
|
+
```
|
19
|
+
|
20
|
+
|
21
|
+
### Create credentials
|
22
|
+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/gitlab)
|
23
|
+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.yaml` file.
|
24
|
+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
|
25
|
+
See `sample_files/sample_config.json` for a sample config file.
|
26
|
+
|
27
|
+
|
28
|
+
### Locally running the connector
|
29
|
+
```
|
30
|
+
poetry run source-gitlab spec
|
31
|
+
poetry run source-gitlab check --config secrets/config.json
|
32
|
+
poetry run source-gitlab discover --config secrets/config.json
|
33
|
+
poetry run source-gitlab read --config secrets/config.json --catalog sample_files/configured_catalog.json
|
34
|
+
```
|
35
|
+
|
36
|
+
### Running unit tests
|
37
|
+
To run unit tests locally, from the connector directory run:
|
38
|
+
```
|
39
|
+
poetry run pytest unit_tests
|
40
|
+
```
|
41
|
+
|
42
|
+
### Building the docker image
|
43
|
+
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
|
44
|
+
2. Run the following command to build the docker image:
|
45
|
+
```bash
|
46
|
+
airbyte-ci connectors --name=source-gitlab build
|
47
|
+
```
|
48
|
+
|
49
|
+
An image will be available on your host with the tag `airbyte/source-gitlab:dev`.
|
50
|
+
|
51
|
+
|
52
|
+
### Running as a docker container
|
53
|
+
Then run any of the connector commands as follows:
|
54
|
+
```
|
55
|
+
docker run --rm airbyte/source-gitlab:dev spec
|
56
|
+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
|
57
|
+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
|
58
|
+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
|
59
|
+
```
|
60
|
+
|
61
|
+
### Running our CI test suite
|
62
|
+
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
|
63
|
+
```bash
|
64
|
+
airbyte-ci connectors --name=source-gitlab test
|
65
|
+
```
|
66
|
+
|
67
|
+
### Customizing acceptance Tests
|
68
|
+
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
|
69
|
+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
|
70
|
+
|
71
|
+
### Dependency Management
|
72
|
+
All of your dependencies should be managed via Poetry.
|
73
|
+
To add a new dependency, run:
|
74
|
+
```bash
|
75
|
+
poetry add <package-name>
|
76
|
+
```
|
77
|
+
|
78
|
+
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
|
79
|
+
|
80
|
+
## Publishing a new version of the connector
|
81
|
+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
|
82
|
+
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
|
83
|
+
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
|
84
|
+
- bump the `dockerImageTag` value in in `metadata.yaml`
|
85
|
+
- bump the `version` value in `pyproject.toml`
|
86
|
+
3. Make sure the `metadata.yaml` content is up to date.
|
87
|
+
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
|
88
|
+
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
|
89
|
+
6. Pat yourself on the back for being an awesome contributor.
|
90
|
+
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|
91
|
+
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
|
@@ -0,0 +1,34 @@
|
|
1
|
+
[build-system]
|
2
|
+
requires = [
|
3
|
+
"poetry-core>=1.0.0",
|
4
|
+
]
|
5
|
+
build-backend = "poetry.core.masonry.api"
|
6
|
+
|
7
|
+
[tool.poetry]
|
8
|
+
version = "3.0.0"
|
9
|
+
name = "airbyte-source-gitlab"
|
10
|
+
description = "Source implementation for Gitlab."
|
11
|
+
authors = [
|
12
|
+
"Airbyte <contact@airbyte.io>",
|
13
|
+
]
|
14
|
+
license = "MIT"
|
15
|
+
readme = "README.md"
|
16
|
+
documentation = "https://docs.airbyte.com/integrations/sources/gitlab"
|
17
|
+
homepage = "https://airbyte.com"
|
18
|
+
repository = "https://github.com/airbytehq/airbyte"
|
19
|
+
packages = [
|
20
|
+
{ include = "source_gitlab" },
|
21
|
+
]
|
22
|
+
|
23
|
+
[tool.poetry.dependencies]
|
24
|
+
python = "^3.9,<3.12"
|
25
|
+
airbyte-cdk = "==0.58.8"
|
26
|
+
vcrpy = "==4.1.1"
|
27
|
+
|
28
|
+
[tool.poetry.scripts]
|
29
|
+
source-gitlab = "source_gitlab.run:run"
|
30
|
+
|
31
|
+
[tool.poetry.group.dev.dependencies]
|
32
|
+
pytest-mock = "^3.12.0"
|
33
|
+
requests-mock = "^1.9.3"
|
34
|
+
pytest = "^6.1"
|
{airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/config_migrations.py
RENAMED
@@ -95,12 +95,10 @@ class MigrateStringToArray(ABC):
|
|
95
95
|
|
96
96
|
|
97
97
|
class MigrateGroups(MigrateStringToArray):
|
98
|
-
|
99
98
|
migrate_from_key: str = "groups"
|
100
99
|
migrate_to_key: str = "groups_list"
|
101
100
|
|
102
101
|
|
103
102
|
class MigrateProjects(MigrateStringToArray):
|
104
|
-
|
105
103
|
migrate_from_key: str = "projects"
|
106
104
|
migrate_to_key: str = "projects_list"
|
{airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/groups.json
RENAMED
@@ -38,6 +38,17 @@
|
|
38
38
|
}
|
39
39
|
}
|
40
40
|
}
|
41
|
+
},
|
42
|
+
"allowed_to_push": {
|
43
|
+
"type": ["null", "array"],
|
44
|
+
"items": {
|
45
|
+
"type": ["null", "object"],
|
46
|
+
"properties": {
|
47
|
+
"access_level": {
|
48
|
+
"type": ["null", "integer"]
|
49
|
+
}
|
50
|
+
}
|
51
|
+
}
|
41
52
|
}
|
42
53
|
}
|
43
54
|
},
|
@@ -74,6 +85,9 @@
|
|
74
85
|
"subgroup_creation_level": {
|
75
86
|
"type": ["null", "string"]
|
76
87
|
},
|
88
|
+
"enabled_git_access_protocol": {
|
89
|
+
"type": ["null", "string"]
|
90
|
+
},
|
77
91
|
"emails_disabled": {
|
78
92
|
"type": ["null", "boolean"]
|
79
93
|
},
|
@@ -153,6 +167,12 @@
|
|
153
167
|
},
|
154
168
|
"service_access_tokens_expiration_enforced": {
|
155
169
|
"type": ["null", "boolean"]
|
170
|
+
},
|
171
|
+
"lock_math_rendering_limits_enabled": {
|
172
|
+
"type": ["null", "boolean"]
|
173
|
+
},
|
174
|
+
"math_rendering_limits_enabled": {
|
175
|
+
"type": ["null", "boolean"]
|
156
176
|
}
|
157
177
|
}
|
158
178
|
}
|
{airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/issues.json
RENAMED
@@ -118,6 +118,9 @@
|
|
118
118
|
},
|
119
119
|
"id": {
|
120
120
|
"type": ["null", "integer"]
|
121
|
+
},
|
122
|
+
"locked": {
|
123
|
+
"type": ["null", "boolean"]
|
121
124
|
}
|
122
125
|
}
|
123
126
|
},
|
@@ -141,6 +144,12 @@
|
|
141
144
|
},
|
142
145
|
"username": {
|
143
146
|
"type": ["null", "string"]
|
147
|
+
},
|
148
|
+
"id": {
|
149
|
+
"type": ["null", "integer"]
|
150
|
+
},
|
151
|
+
"locked": {
|
152
|
+
"type": ["null", "boolean"]
|
144
153
|
}
|
145
154
|
}
|
146
155
|
},
|
@@ -164,6 +173,18 @@
|
|
164
173
|
},
|
165
174
|
"username": {
|
166
175
|
"type": ["null", "string"]
|
176
|
+
},
|
177
|
+
"human_readable_end_date": {
|
178
|
+
"type": ["null", "string"]
|
179
|
+
},
|
180
|
+
"human_readable_timestamp": {
|
181
|
+
"type": ["null", "string"]
|
182
|
+
},
|
183
|
+
"id": {
|
184
|
+
"type": ["null", "integer"]
|
185
|
+
},
|
186
|
+
"locked": {
|
187
|
+
"type": ["null", "boolean"]
|
167
188
|
}
|
168
189
|
}
|
169
190
|
},
|
@@ -251,6 +272,15 @@
|
|
251
272
|
},
|
252
273
|
"group_id": {
|
253
274
|
"type": ["null", "integer"]
|
275
|
+
},
|
276
|
+
"locked": {
|
277
|
+
"type": ["null", "boolean"]
|
278
|
+
},
|
279
|
+
"human_readable_end_date": {
|
280
|
+
"type": ["null", "string"]
|
281
|
+
},
|
282
|
+
"human_readable_timestamp": {
|
283
|
+
"type": ["null", "string"]
|
254
284
|
}
|
255
285
|
}
|
256
286
|
},
|
@@ -0,0 +1,85 @@
|
|
1
|
+
{
|
2
|
+
"$schema": "https://json-schema.org/draft-07/schema#",
|
3
|
+
"type": "object",
|
4
|
+
"properties": {
|
5
|
+
"project_id": {
|
6
|
+
"type": ["null", "integer"]
|
7
|
+
},
|
8
|
+
"id": {
|
9
|
+
"type": ["null", "string"]
|
10
|
+
},
|
11
|
+
"short_id": {
|
12
|
+
"type": ["null", "string"]
|
13
|
+
},
|
14
|
+
"created_at": {
|
15
|
+
"type": ["null", "string"],
|
16
|
+
"format": "date-time"
|
17
|
+
},
|
18
|
+
"parent_ids": {
|
19
|
+
"type": ["null", "array"],
|
20
|
+
"items": {
|
21
|
+
"type": ["null", "string"]
|
22
|
+
}
|
23
|
+
},
|
24
|
+
"title": {
|
25
|
+
"type": ["null", "string"]
|
26
|
+
},
|
27
|
+
"message": {
|
28
|
+
"type": ["null", "string"]
|
29
|
+
},
|
30
|
+
"author_name": {
|
31
|
+
"type": ["null", "string"]
|
32
|
+
},
|
33
|
+
"author_email": {
|
34
|
+
"type": ["null", "string"]
|
35
|
+
},
|
36
|
+
"authored_date": {
|
37
|
+
"type": ["null", "string"],
|
38
|
+
"format": "date-time"
|
39
|
+
},
|
40
|
+
"extended_trailers": {
|
41
|
+
"type": ["null", "object"],
|
42
|
+
"properties": {
|
43
|
+
"Cc": {
|
44
|
+
"type": ["null", "array"],
|
45
|
+
"items": {
|
46
|
+
"type": ["null", "string"]
|
47
|
+
}
|
48
|
+
}
|
49
|
+
}
|
50
|
+
},
|
51
|
+
"committer_name": {
|
52
|
+
"type": ["null", "string"]
|
53
|
+
},
|
54
|
+
"committer_email": {
|
55
|
+
"type": ["null", "string"]
|
56
|
+
},
|
57
|
+
"committed_date": {
|
58
|
+
"type": ["null", "string"],
|
59
|
+
"format": "date-time"
|
60
|
+
},
|
61
|
+
"trailers": {
|
62
|
+
"type": ["null", "object"]
|
63
|
+
},
|
64
|
+
"web_url": {
|
65
|
+
"type": ["null", "string"]
|
66
|
+
},
|
67
|
+
"stats": {
|
68
|
+
"type": ["null", "object"],
|
69
|
+
"properties": {
|
70
|
+
"additions": {
|
71
|
+
"type": ["null", "integer"]
|
72
|
+
},
|
73
|
+
"deletions": {
|
74
|
+
"type": ["null", "integer"]
|
75
|
+
},
|
76
|
+
"total": {
|
77
|
+
"type": ["null", "integer"]
|
78
|
+
}
|
79
|
+
}
|
80
|
+
},
|
81
|
+
"merge_request_iid": {
|
82
|
+
"type": ["null", "integer"]
|
83
|
+
}
|
84
|
+
}
|
85
|
+
}
|
{airbyte-source-gitlab-2.1.1 → airbyte_source_gitlab-3.0.0}/source_gitlab/schemas/projects.json
RENAMED
@@ -123,6 +123,9 @@
|
|
123
123
|
},
|
124
124
|
"members": {
|
125
125
|
"type": ["null", "string"]
|
126
|
+
},
|
127
|
+
"cluster_agents": {
|
128
|
+
"type": ["null", "string"]
|
126
129
|
}
|
127
130
|
}
|
128
131
|
},
|
@@ -329,6 +332,15 @@
|
|
329
332
|
},
|
330
333
|
"packages_size": {
|
331
334
|
"type": ["null", "integer"]
|
335
|
+
},
|
336
|
+
"container_registry_size": {
|
337
|
+
"type": ["null", "integer"]
|
338
|
+
},
|
339
|
+
"pipeline_artifacts_size": {
|
340
|
+
"type": ["null", "integer"]
|
341
|
+
},
|
342
|
+
"uploads_size": {
|
343
|
+
"type": ["null", "integer"]
|
332
344
|
}
|
333
345
|
}
|
334
346
|
},
|
@@ -492,6 +504,12 @@
|
|
492
504
|
},
|
493
505
|
"ci_restrict_pipeline_cancellation_role": {
|
494
506
|
"type": ["null", "string"]
|
507
|
+
},
|
508
|
+
"repository_object_format": {
|
509
|
+
"type": ["null", "string"]
|
510
|
+
},
|
511
|
+
"warn_about_potentially_unwanted_characters": {
|
512
|
+
"type": ["null", "boolean"]
|
495
513
|
}
|
496
514
|
}
|
497
515
|
}
|
@@ -339,8 +339,10 @@ class MergeRequests(IncrementalGitlabChildStream):
|
|
339
339
|
|
340
340
|
|
341
341
|
class MergeRequestCommits(GitlabChildStream):
|
342
|
+
"""Docs: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-merge-request-commits"""
|
343
|
+
|
342
344
|
path_list = ["project_id", "iid"]
|
343
|
-
path_template = "projects/{project_id}/merge_requests/{iid}"
|
345
|
+
path_template = "projects/{project_id}/merge_requests/{iid}/commits"
|
344
346
|
|
345
347
|
def transform(self, record, stream_slice: Mapping[str, Any] = None, **kwargs):
|
346
348
|
super().transform(record, stream_slice, **kwargs)
|
@@ -1,141 +0,0 @@
|
|
1
|
-
Metadata-Version: 2.1
|
2
|
-
Name: airbyte-source-gitlab
|
3
|
-
Version: 2.1.1
|
4
|
-
Summary: Source implementation for Gitlab.
|
5
|
-
Author: Airbyte
|
6
|
-
Author-email: contact@airbyte.io
|
7
|
-
Description-Content-Type: text/markdown
|
8
|
-
Requires-Dist: airbyte-cdk
|
9
|
-
Requires-Dist: vcrpy==4.1.1
|
10
|
-
Provides-Extra: tests
|
11
|
-
Requires-Dist: requests-mock~=1.9.3; extra == "tests"
|
12
|
-
Requires-Dist: pytest~=6.1; extra == "tests"
|
13
|
-
Requires-Dist: requests_mock; extra == "tests"
|
14
|
-
Requires-Dist: pytest-mock; extra == "tests"
|
15
|
-
|
16
|
-
# Gitlab Source
|
17
|
-
|
18
|
-
This is the repository for the Gitlab source connector, written in Python.
|
19
|
-
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/gitlab).
|
20
|
-
|
21
|
-
|
22
|
-
**To iterate on this connector, make sure to complete this prerequisites section.**
|
23
|
-
|
24
|
-
From this connector directory, create a virtual environment:
|
25
|
-
```
|
26
|
-
python -m venv .venv
|
27
|
-
```
|
28
|
-
|
29
|
-
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
|
30
|
-
development environment of choice. To activate it from the terminal, run:
|
31
|
-
```
|
32
|
-
source .venv/bin/activate
|
33
|
-
pip install -r requirements.txt
|
34
|
-
```
|
35
|
-
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
|
36
|
-
|
37
|
-
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
|
38
|
-
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
|
39
|
-
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
|
40
|
-
should work as you expect.
|
41
|
-
|
42
|
-
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/gitlab)
|
43
|
-
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_gitlab/spec.json` file.
|
44
|
-
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
|
45
|
-
See `integration_tests/sample_config.json` for a sample config file.
|
46
|
-
|
47
|
-
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source gitlab test creds`
|
48
|
-
and place them into `secrets/config.json`.
|
49
|
-
|
50
|
-
```
|
51
|
-
python main.py spec
|
52
|
-
python main.py check --config secrets/config.json
|
53
|
-
python main.py discover --config secrets/config.json
|
54
|
-
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
|
55
|
-
```
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
|
61
|
-
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
|
62
|
-
Then running the following command will build your connector:
|
63
|
-
|
64
|
-
```bash
|
65
|
-
airbyte-ci connectors --name=source-gitlab build
|
66
|
-
```
|
67
|
-
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-gitlab:dev`.
|
68
|
-
|
69
|
-
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
|
70
|
-
You can customize our build process by adding a `build_customization.py` module to your connector.
|
71
|
-
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
|
72
|
-
It will be imported at runtime by our build process and the functions will be called if they exist.
|
73
|
-
|
74
|
-
Here is an example of a `build_customization.py` module:
|
75
|
-
```python
|
76
|
-
from __future__ import annotations
|
77
|
-
|
78
|
-
from typing import TYPE_CHECKING
|
79
|
-
|
80
|
-
if TYPE_CHECKING:
|
81
|
-
from dagger import Container
|
82
|
-
|
83
|
-
|
84
|
-
async def pre_connector_install(base_image_container: Container) -> Container:
|
85
|
-
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
|
86
|
-
|
87
|
-
async def post_connector_install(connector_container: Container) -> Container:
|
88
|
-
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
|
89
|
-
```
|
90
|
-
|
91
|
-
This connector is built using our dynamic built process in `airbyte-ci`.
|
92
|
-
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
|
93
|
-
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
|
94
|
-
It does not rely on a Dockerfile.
|
95
|
-
|
96
|
-
If you would like to patch our connector and build your own a simple approach would be to:
|
97
|
-
|
98
|
-
1. Create your own Dockerfile based on the latest version of the connector image.
|
99
|
-
```Dockerfile
|
100
|
-
FROM airbyte/source-gitlab:latest
|
101
|
-
|
102
|
-
COPY . ./airbyte/integration_code
|
103
|
-
RUN pip install ./airbyte/integration_code
|
104
|
-
|
105
|
-
```
|
106
|
-
Please use this as an example. This is not optimized.
|
107
|
-
|
108
|
-
2. Build your image:
|
109
|
-
```bash
|
110
|
-
docker build -t airbyte/source-gitlab:dev .
|
111
|
-
docker run airbyte/source-gitlab:dev spec
|
112
|
-
```
|
113
|
-
Then run any of the connector commands as follows:
|
114
|
-
```
|
115
|
-
docker run --rm airbyte/source-gitlab:dev spec
|
116
|
-
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev check --config /secrets/config.json
|
117
|
-
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-gitlab:dev discover --config /secrets/config.json
|
118
|
-
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-gitlab:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
|
119
|
-
```
|
120
|
-
|
121
|
-
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
|
122
|
-
```bash
|
123
|
-
airbyte-ci connectors --name=source-gitlab test
|
124
|
-
```
|
125
|
-
|
126
|
-
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
|
127
|
-
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
|
128
|
-
|
129
|
-
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
|
130
|
-
We split dependencies between two groups, dependencies that are:
|
131
|
-
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
|
132
|
-
* required for the testing need to go to `TEST_REQUIREMENTS` list
|
133
|
-
|
134
|
-
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
|
135
|
-
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-gitlab test`
|
136
|
-
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
|
137
|
-
3. Make sure the `metadata.yaml` content is up to date.
|
138
|
-
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/gitlab.md`).
|
139
|
-
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
|
140
|
-
6. Pat yourself on the back for being an awesome contributor.
|
141
|
-
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|