trellodon 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +27 -1
- data/README.md +91 -2
- data/lib/{trellodon/api_executor.rb → .rbnext/2.7/trellodon/executor.rb} +6 -9
- data/lib/.rbnext/3.0/trellodon/client.rb +149 -0
- data/lib/.rbnext/3.1/trellodon/schedulers/concurrent.rb +30 -0
- data/lib/trellodon/cli.rb +68 -17
- data/lib/trellodon/{api_client.rb → client.rb} +53 -8
- data/lib/trellodon/entities/attachment.rb +1 -1
- data/lib/trellodon/entities/card.rb +1 -1
- data/lib/trellodon/entities/checklist.rb +5 -0
- data/lib/trellodon/entities/checklist_item.rb +9 -0
- data/lib/trellodon/entities/comment.rb +1 -1
- data/lib/trellodon/entities/member.rb +5 -0
- data/lib/trellodon/entities.rb +3 -0
- data/lib/trellodon/executor.rb +65 -0
- data/lib/trellodon/formatters/markdown.rb +62 -11
- data/lib/trellodon/schedulers/concurrent.rb +30 -0
- data/lib/trellodon/schedulers.rb +1 -0
- data/lib/trellodon/version.rb +1 -1
- data/lib/trellodon.rb +7 -1
- metadata +41 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a61911e92509a77f13aa90f9f56c14d038dbe443b4e1063d28f66b6745b04432
|
4
|
+
data.tar.gz: '079609a773107e786a7d795fe7d23564ded2aa66e8d587a6fd22b4e288e15478'
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 617119b48c1f2385c87d57ffd2683933097ed1aae11a51577f65e16012ab1f58b65187f57884123ed04ba522797925826e3cf202e7d4dad9d1fd63a3d736bcef
|
7
|
+
data.tar.gz: 30c1f1bcb2df2c69a3c8d0cebc93a597f16d2207d33bef979670134d306bd0173cc97948f38f739cf1d6b5eef0a7a602112fb0de40021d078d0188280ca4d039
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,31 @@
|
|
1
|
-
|
1
|
+
# Change log
|
2
|
+
|
3
|
+
## [master]
|
4
|
+
|
5
|
+
## [0.3.0] - 2022-04-12 🚀🚀🚀
|
6
|
+
|
7
|
+
- Handle missing members. ([@palkan][])
|
8
|
+
|
9
|
+
- Use inline scheduler by default. ([@palkan][])
|
10
|
+
|
11
|
+
- Replace `--sync` with `--concurrency=0`. ([@palkan][])
|
12
|
+
|
13
|
+
## [0.2.1] - 2022-04-12 🚀
|
14
|
+
|
15
|
+
- Misc changes.
|
16
|
+
|
17
|
+
## [0.2.0] - 2022-04-12
|
18
|
+
|
19
|
+
- Store Trello API creds locally for avoid asking it before execution ([@fargelus][]).
|
20
|
+
- Download attachments, human names in comments, rename index.md to README.md ([@rinasergeeva][]).
|
21
|
+
- Added minimal project README.md ([@rinasergeeva][], [@fargelus][]).
|
22
|
+
- --sync param for switch to `Schedulers::Inline` in CLI ([@fargelus][]).
|
23
|
+
- Concurrent scheduler with customizable thread pool size ([@fargelus][]).
|
24
|
+
- Checklists support ([@rinasergeeva][]).
|
2
25
|
|
3
26
|
## [0.1.0] - 2022-04-07
|
4
27
|
|
5
28
|
- Initial release
|
29
|
+
|
30
|
+
[@fargelus]: https://github.com/fargelus
|
31
|
+
[@rinasergeeva]: https://github.com/rinasergeeva
|
data/README.md
CHANGED
@@ -1,3 +1,92 @@
|
|
1
|
-
|
1
|
+

|
2
|
+
[](https://rubygems.org/gems/trellodon)
|
2
3
|
|
3
|
-
|
4
|
+
# Trellodon
|
5
|
+
|
6
|
+
A Ruby tool to export a Trello board and convert it into a set of folders and markdown files, corresponding to lists and cards on the board. For each card, you’ll get the basic details such as:
|
7
|
+
|
8
|
+
* Name
|
9
|
+
* Description
|
10
|
+
* Labels
|
11
|
+
* Comments
|
12
|
+
* Attachments
|
13
|
+
|
14
|
+
<img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54"></a>
|
15
|
+
|
16
|
+
## Installation
|
17
|
+
|
18
|
+
```
|
19
|
+
gem install trellodon
|
20
|
+
```
|
21
|
+
|
22
|
+
## Prerequisites
|
23
|
+
|
24
|
+
Trellodon needs two secret codes to download your boards:
|
25
|
+
|
26
|
+
* Trello API key
|
27
|
+
* Authentication token
|
28
|
+
|
29
|
+
To generate them, go to the [trello.com/app-key](https://trello.com/app-key) page and follow the instructions.
|
30
|
+
|
31
|
+
## Usage
|
32
|
+
|
33
|
+
```sh
|
34
|
+
Usage: trellodon dump [options]
|
35
|
+
Options:
|
36
|
+
--board VALUE Board URL or ID
|
37
|
+
--out VALUE Destination folder path
|
38
|
+
--concurrency VALUE Amount of processing threads (default: 4). Set to 0 to execute API requests in-line
|
39
|
+
--clear-auth Remove saved api credentials
|
40
|
+
-v, --version Print current version
|
41
|
+
-h, --help Print help
|
42
|
+
```
|
43
|
+
|
44
|
+
### Detailed example
|
45
|
+
|
46
|
+
Suppose we have a Trello board called `projects` with lists `Brainstorm`, `TODO`, `DOING`, `DONE`. Lists contains own cards.<br>
|
47
|
+
All Trello boards have its own id this id requires Trellodon to fetch board via Trello API.
|
48
|
+
You can find this id in two ways:
|
49
|
+
|
50
|
+
* In URL `https://trello.com/b/{id}/projects` your Trello board id will be placed as `{id}` in example;
|
51
|
+
* Put `.json` in your board URL like this `https://trello.com/b/{id}/projects.json` id field in json output is what you need;
|
52
|
+
|
53
|
+
You can also paste whole board URL and Trellodon will parse it correctly.<br>
|
54
|
+
After launch Trellodon creates follow output in specified folder:
|
55
|
+
|
56
|
+
```sh
|
57
|
+
projects/
|
58
|
+
Brainstorm/
|
59
|
+
first_card_title/
|
60
|
+
README.md
|
61
|
+
attachments/
|
62
|
+
image.png
|
63
|
+
TODO/
|
64
|
+
first_card_title/
|
65
|
+
README.md
|
66
|
+
attachments/
|
67
|
+
report.docx
|
68
|
+
...
|
69
|
+
```
|
70
|
+
|
71
|
+
Each card has follow output format:
|
72
|
+
|
73
|
+
```md
|
74
|
+
---
|
75
|
+
title: card_title
|
76
|
+
last_updated_at: 2022-03-16 16:28:39 UTC
|
77
|
+
labels: Test
|
78
|
+
---
|
79
|
+
|
80
|
+
# card_title
|
81
|
+
|
82
|
+
## Description
|
83
|
+
Some card description
|
84
|
+
|
85
|
+
## Comments
|
86
|
+
** John Doe @doe at 2022-03-16 16:28:39 UTC**
|
87
|
+
|
88
|
+
## Attachments
|
89
|
+
### Image.png
|
90
|
+
**date**: 2022-03-16
|
91
|
+
**url**: https://trello.com/1/cards/#{card_id}/attachments/#{attachment.id}/download/image.png
|
92
|
+
```
|
@@ -1,10 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require "trellodon/api_client"
|
4
|
-
require "trellodon/config"
|
5
|
-
|
6
3
|
module Trellodon
|
7
|
-
class
|
4
|
+
class Executor
|
8
5
|
BOARD_ID_REGEX = /\/b\/([^\/]+)\//
|
9
6
|
private_constant :BOARD_ID_REGEX
|
10
7
|
|
@@ -16,7 +13,7 @@ module Trellodon
|
|
16
13
|
@scheduler = scheduler
|
17
14
|
check_credentials!
|
18
15
|
|
19
|
-
@
|
16
|
+
@client = Client.new(api_key: @api_key, api_token: @api_token)
|
20
17
|
end
|
21
18
|
|
22
19
|
def download(board_pointer)
|
@@ -25,13 +22,13 @@ module Trellodon
|
|
25
22
|
startup_time = Time.now
|
26
23
|
logger.debug "Fetching board 🚀️️"
|
27
24
|
board = scheduler.post do
|
28
|
-
|
25
|
+
client.fetch_board(board_id).tap { |_1| formatter.board_added(_1) }
|
29
26
|
end.value
|
30
27
|
|
31
|
-
logger.debug "Fetching cards
|
28
|
+
logger.debug "Fetching #{board.card_ids.size} cards from the board with comments and attachments 🐢"
|
32
29
|
board.card_ids.map do |card_id|
|
33
30
|
scheduler.post do
|
34
|
-
|
31
|
+
client.fetch_card(card_id).tap { |_1| formatter.card_added(_1) }
|
35
32
|
end
|
36
33
|
end.map(&:value)
|
37
34
|
|
@@ -41,7 +38,7 @@ module Trellodon
|
|
41
38
|
|
42
39
|
private
|
43
40
|
|
44
|
-
attr_reader :
|
41
|
+
attr_reader :client, :board_id, :logger, :scheduler, :formatter
|
45
42
|
|
46
43
|
def check_credentials!
|
47
44
|
return if @api_key.to_s.size.positive? && @api_token.to_s.size.positive?
|
@@ -0,0 +1,149 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trello"
|
4
|
+
require "trellodon/entities"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
class Client
|
8
|
+
class DeletedMember < Struct.new(:id)
|
9
|
+
def full_name ; "[DELETED]"; end
|
10
|
+
|
11
|
+
def username ; "__deleted__"; end
|
12
|
+
end
|
13
|
+
|
14
|
+
def initialize(api_key:, api_token:, logger: Config.logger)
|
15
|
+
@logger = logger
|
16
|
+
@client = Trello::Client.new(
|
17
|
+
developer_public_key: api_key,
|
18
|
+
member_token: api_token
|
19
|
+
)
|
20
|
+
@members = {}
|
21
|
+
end
|
22
|
+
|
23
|
+
def fetch_board(board_id)
|
24
|
+
retrying do
|
25
|
+
board = @client.find(:boards, board_id)
|
26
|
+
|
27
|
+
Board.new(
|
28
|
+
id: board.id,
|
29
|
+
short_id: board.short_url.chars.last(8).join,
|
30
|
+
name: board.name,
|
31
|
+
lists: lists_from(board),
|
32
|
+
card_ids: @client.find_many(Trello::Card, "/boards/#{board_id}/cards", fields: "id").map(&:id),
|
33
|
+
last_activity_date: board.last_activity_date
|
34
|
+
)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
def fetch_card(card_id)
|
39
|
+
retrying do
|
40
|
+
card = @client.find(:cards, card_id)
|
41
|
+
|
42
|
+
Card.new(
|
43
|
+
id: card.id,
|
44
|
+
short_id: card.short_url.chars.last(8).join,
|
45
|
+
name: card.name,
|
46
|
+
desc: card.desc,
|
47
|
+
labels: card.labels.map(&:name),
|
48
|
+
list_id: card.list_id,
|
49
|
+
comments: comments_from(card),
|
50
|
+
attachments: attachments_from(card),
|
51
|
+
checklists: checklists_from(card),
|
52
|
+
last_activity_date: card.last_activity_date
|
53
|
+
)
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
private
|
58
|
+
|
59
|
+
attr_reader :logger
|
60
|
+
|
61
|
+
def retrying(&block)
|
62
|
+
attempt = 0
|
63
|
+
begin
|
64
|
+
block.call
|
65
|
+
rescue Trello::Error => err
|
66
|
+
raise unless err.status_code == 429
|
67
|
+
attempt += 1
|
68
|
+
cooldown = 2**attempt + rand(2**attempt) - 2**(attempt - 1)
|
69
|
+
logger.warn "API limit exceeded, cool down for #{cooldown}s"
|
70
|
+
sleep cooldown
|
71
|
+
retry
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
def comments_from(card)
|
76
|
+
card.comments.map do |comment|
|
77
|
+
Comment.new(
|
78
|
+
text: comment.data["text"],
|
79
|
+
date: comment.date,
|
80
|
+
creator: member_from(comment)
|
81
|
+
)
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
def attachments_from(card)
|
86
|
+
card.attachments.map do |attach|
|
87
|
+
Attachment.new(
|
88
|
+
file_name: attach.file_name,
|
89
|
+
mime_type: attach.mime_type,
|
90
|
+
bytes: attach.bytes,
|
91
|
+
date: attach.date,
|
92
|
+
name: attach.name,
|
93
|
+
is_upload: attach.is_upload,
|
94
|
+
headers: attach.is_upload ? {"Authorization" => "OAuth oauth_consumer_key=\"#{@client.configuration.developer_public_key}\", oauth_token=\"#{@client.configuration.member_token}\""} : {},
|
95
|
+
url: attach.url
|
96
|
+
)
|
97
|
+
end
|
98
|
+
end
|
99
|
+
|
100
|
+
def lists_from(board)
|
101
|
+
board.lists.map do |list|
|
102
|
+
List.new(
|
103
|
+
id: list.id,
|
104
|
+
name: list.name,
|
105
|
+
short_id: board.short_url.chars.last(8).join
|
106
|
+
)
|
107
|
+
end
|
108
|
+
end
|
109
|
+
|
110
|
+
def checklists_from(card)
|
111
|
+
card.checklists.map do |checklist|
|
112
|
+
Checklist.new(
|
113
|
+
id: checklist.id,
|
114
|
+
name: checklist.name,
|
115
|
+
items: checklist_items_from(checklist)
|
116
|
+
)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
def checklist_items_from(checklist)
|
121
|
+
checklist.items.map do |item|
|
122
|
+
ChecklistItem.new(
|
123
|
+
id: item.id,
|
124
|
+
name: item.name,
|
125
|
+
state: item.state
|
126
|
+
)
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
def member_from(comment)
|
131
|
+
return @members[comment.creator_id] if @members.has_key? comment.creator_id
|
132
|
+
|
133
|
+
member =
|
134
|
+
begin
|
135
|
+
@client.find(:members, comment.creator_id)
|
136
|
+
rescue Trello::Error => err
|
137
|
+
raise unless err.status_code == 404
|
138
|
+
|
139
|
+
DeletedMember.new(comment.creator_id)
|
140
|
+
end
|
141
|
+
|
142
|
+
@members[comment.creator_id] = Member.new(
|
143
|
+
id: member.id,
|
144
|
+
full_name: member.full_name,
|
145
|
+
username: member.username
|
146
|
+
)
|
147
|
+
end
|
148
|
+
end
|
149
|
+
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trellodon/schedulers/base"
|
4
|
+
require "concurrent"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
module Schedulers
|
8
|
+
class Concurrent < Base
|
9
|
+
using RubyNext
|
10
|
+
|
11
|
+
DEFAULT_MAX_THREADS = 4
|
12
|
+
|
13
|
+
attr_reader :max_threads
|
14
|
+
|
15
|
+
def initialize(threads_count = DEFAULT_MAX_THREADS)
|
16
|
+
@max_threads = threads_count
|
17
|
+
end
|
18
|
+
|
19
|
+
def post(&block)
|
20
|
+
::Concurrent::Future.execute(executor: executor, &block)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def executor
|
26
|
+
::Concurrent::FixedThreadPool.new(max_threads)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/trellodon/cli.rb
CHANGED
@@ -1,36 +1,34 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "trellodon"
|
4
|
-
|
5
4
|
require "tty-prompt"
|
5
|
+
require "fileutils"
|
6
6
|
|
7
7
|
module Trellodon
|
8
8
|
class CLI
|
9
|
-
class Prompt
|
10
|
-
|
11
|
-
@prompt = TTY::Prompt.new
|
12
|
-
end
|
9
|
+
class Prompt < TTY::Prompt
|
10
|
+
include Singleton
|
13
11
|
|
14
12
|
def ask_api_key
|
15
|
-
|
13
|
+
mask(
|
16
14
|
"Provide your Developer API Key (see https://trello.com/app-key):"
|
17
15
|
) { |q| q.required true }
|
18
16
|
end
|
19
17
|
|
20
18
|
def ask_api_token
|
21
|
-
|
19
|
+
mask(
|
22
20
|
"Provide your API token:"
|
23
21
|
) { |q| q.required true }
|
24
22
|
end
|
25
23
|
|
26
24
|
def ask_board
|
27
|
-
|
25
|
+
ask("Which board would you like to dump? (URL or ID)") do |q|
|
28
26
|
q.required true
|
29
27
|
end
|
30
28
|
end
|
31
29
|
|
32
30
|
def ask_folder
|
33
|
-
|
31
|
+
ask("Destination folder?", default: "./")
|
34
32
|
end
|
35
33
|
end
|
36
34
|
|
@@ -44,10 +42,15 @@ module Trellodon
|
|
44
42
|
out: "Destination folder path"
|
45
43
|
)
|
46
44
|
|
47
|
-
extend_options do |opts,
|
48
|
-
opts.banner = "Usage: trellodon dump\n"\
|
45
|
+
extend_options do |opts, config|
|
46
|
+
opts.banner = "Usage: trellodon dump \e[34m[options]\e[0m\n"\
|
49
47
|
"Options:\n"
|
50
48
|
|
49
|
+
opts.on("--concurrency VALUE", "Amount of processing threads (default: 4)")
|
50
|
+
opts.on("--clear-auth", "Remove saved api credentials") do
|
51
|
+
config.reload(api_token: nil, api_key: nil)
|
52
|
+
end
|
53
|
+
|
51
54
|
opts.on_tail("-v", "--version", "Print current version") do
|
52
55
|
puts Trellodon::VERSION
|
53
56
|
exit
|
@@ -59,8 +62,25 @@ module Trellodon
|
|
59
62
|
end
|
60
63
|
end
|
61
64
|
|
65
|
+
def resolve_config_path(_, _)
|
66
|
+
config_path
|
67
|
+
end
|
68
|
+
|
69
|
+
def config_path
|
70
|
+
File.join(Dir.home, ".config", "#{config_name}.yml")
|
71
|
+
end
|
72
|
+
|
73
|
+
def update_config_file!
|
74
|
+
dir = File.dirname(config_path)
|
75
|
+
FileUtils.mkdir_p(dir) unless File.exist?(dir)
|
76
|
+
|
77
|
+
yml = "api_token: #{api_token}\n"\
|
78
|
+
"api_key: #{api_key}"
|
79
|
+
File.write(config_path, yml)
|
80
|
+
end
|
81
|
+
|
62
82
|
private def prompt
|
63
|
-
|
83
|
+
Prompt.instance
|
64
84
|
end
|
65
85
|
|
66
86
|
def api_key
|
@@ -82,34 +102,65 @@ module Trellodon
|
|
82
102
|
|
83
103
|
def initialize
|
84
104
|
@config = Config.new
|
105
|
+
@scheduler = Schedulers::Inline.new
|
85
106
|
end
|
86
107
|
|
87
108
|
def run
|
88
109
|
check_command!
|
89
110
|
|
90
|
-
|
111
|
+
need_to_fill_config = config.to_h[:api_key].nil?
|
112
|
+
|
113
|
+
executor = Trellodon::Executor.new(
|
91
114
|
api_key: config.api_key,
|
92
115
|
api_token: config.api_token,
|
93
116
|
formatter: Trellodon::Formatters::Markdown.new(output_dir: config.out),
|
94
|
-
scheduler:
|
117
|
+
scheduler: scheduler
|
95
118
|
)
|
119
|
+
|
120
|
+
ask_save_credentials if need_to_fill_config
|
121
|
+
|
96
122
|
executor.download(config.board)
|
97
123
|
end
|
98
124
|
|
99
125
|
private
|
100
126
|
|
101
|
-
attr_reader :config
|
127
|
+
attr_reader :config, :scheduler
|
128
|
+
|
129
|
+
def prompt
|
130
|
+
Prompt.instance
|
131
|
+
end
|
102
132
|
|
103
133
|
def check_command!
|
104
|
-
|
134
|
+
params = {}
|
135
|
+
command, = config.option_parser.parse!(ARGV, into: params)
|
105
136
|
unless command
|
106
137
|
puts config.option_parser.help
|
107
138
|
exit
|
108
139
|
end
|
109
140
|
|
110
|
-
|
141
|
+
if command == "dump"
|
142
|
+
change_scheduler(params) if params.key?(:sync) || params.key?(:concurrency)
|
143
|
+
return
|
144
|
+
end
|
111
145
|
|
112
146
|
raise "Unknown command: #{command}. Available commands: dump"
|
113
147
|
end
|
148
|
+
|
149
|
+
def ask_save_credentials
|
150
|
+
answer = prompt.yes?("Would you like to save api creds so next time trellodon won't asking it?", default: true)
|
151
|
+
config.update_config_file! if answer
|
152
|
+
end
|
153
|
+
|
154
|
+
def change_scheduler(params)
|
155
|
+
return unless params[:concurrency]
|
156
|
+
|
157
|
+
threads = Integer(params[:concurrency])
|
158
|
+
|
159
|
+
return @scheduler = Schedulers::Inline.new if threads.zero?
|
160
|
+
|
161
|
+
raise "Value of concurrency option must be integer and greater than zero" unless threads.positive?
|
162
|
+
|
163
|
+
@scheduler = Schedulers::Concurrent.new(threads)
|
164
|
+
end
|
114
165
|
end
|
115
166
|
end
|
@@ -1,17 +1,23 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "trello"
|
4
|
-
|
5
4
|
require "trellodon/entities"
|
6
5
|
|
7
6
|
module Trellodon
|
8
|
-
class
|
7
|
+
class Client
|
8
|
+
class DeletedMember < Struct.new(:id)
|
9
|
+
def full_name = "[DELETED]"
|
10
|
+
|
11
|
+
def username = "__deleted__"
|
12
|
+
end
|
13
|
+
|
9
14
|
def initialize(api_key:, api_token:, logger: Config.logger)
|
10
15
|
@logger = logger
|
11
16
|
@client = Trello::Client.new(
|
12
17
|
developer_public_key: api_key,
|
13
18
|
member_token: api_token
|
14
19
|
)
|
20
|
+
@members = {}
|
15
21
|
end
|
16
22
|
|
17
23
|
def fetch_board(board_id)
|
@@ -42,6 +48,7 @@ module Trellodon
|
|
42
48
|
list_id: card.list_id,
|
43
49
|
comments: comments_from(card),
|
44
50
|
attachments: attachments_from(card),
|
51
|
+
checklists: checklists_from(card),
|
45
52
|
last_activity_date: card.last_activity_date
|
46
53
|
)
|
47
54
|
end
|
@@ -57,15 +64,10 @@ module Trellodon
|
|
57
64
|
block.call
|
58
65
|
rescue Trello::Error => err
|
59
66
|
raise unless err.status_code == 429
|
60
|
-
|
61
67
|
attempt += 1
|
62
|
-
|
63
68
|
cooldown = 2**attempt + rand(2**attempt) - 2**(attempt - 1)
|
64
|
-
|
65
69
|
logger.warn "API limit exceeded, cool down for #{cooldown}s"
|
66
|
-
|
67
70
|
sleep cooldown
|
68
|
-
|
69
71
|
retry
|
70
72
|
end
|
71
73
|
end
|
@@ -75,7 +77,7 @@ module Trellodon
|
|
75
77
|
Comment.new(
|
76
78
|
text: comment.data["text"],
|
77
79
|
date: comment.date,
|
78
|
-
|
80
|
+
creator: member_from(comment)
|
79
81
|
)
|
80
82
|
end
|
81
83
|
end
|
@@ -86,6 +88,10 @@ module Trellodon
|
|
86
88
|
file_name: attach.file_name,
|
87
89
|
mime_type: attach.mime_type,
|
88
90
|
bytes: attach.bytes,
|
91
|
+
date: attach.date,
|
92
|
+
name: attach.name,
|
93
|
+
is_upload: attach.is_upload,
|
94
|
+
headers: attach.is_upload ? {"Authorization" => "OAuth oauth_consumer_key=\"#{@client.configuration.developer_public_key}\", oauth_token=\"#{@client.configuration.member_token}\""} : {},
|
89
95
|
url: attach.url
|
90
96
|
)
|
91
97
|
end
|
@@ -100,5 +106,44 @@ module Trellodon
|
|
100
106
|
)
|
101
107
|
end
|
102
108
|
end
|
109
|
+
|
110
|
+
def checklists_from(card)
|
111
|
+
card.checklists.map do |checklist|
|
112
|
+
Checklist.new(
|
113
|
+
id: checklist.id,
|
114
|
+
name: checklist.name,
|
115
|
+
items: checklist_items_from(checklist)
|
116
|
+
)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
def checklist_items_from(checklist)
|
121
|
+
checklist.items.map do |item|
|
122
|
+
ChecklistItem.new(
|
123
|
+
id: item.id,
|
124
|
+
name: item.name,
|
125
|
+
state: item.state
|
126
|
+
)
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
def member_from(comment)
|
131
|
+
return @members[comment.creator_id] if @members.has_key? comment.creator_id
|
132
|
+
|
133
|
+
member =
|
134
|
+
begin
|
135
|
+
@client.find(:members, comment.creator_id)
|
136
|
+
rescue Trello::Error => err
|
137
|
+
raise unless err.status_code == 404
|
138
|
+
|
139
|
+
DeletedMember.new(comment.creator_id)
|
140
|
+
end
|
141
|
+
|
142
|
+
@members[comment.creator_id] = Member.new(
|
143
|
+
id: member.id,
|
144
|
+
full_name: member.full_name,
|
145
|
+
username: member.username
|
146
|
+
)
|
147
|
+
end
|
103
148
|
end
|
104
149
|
end
|
@@ -1,5 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module Trellodon
|
4
|
-
Attachment = Struct.new("Attachment", :file_name, :mime_type, :bytes, :url, keyword_init: true)
|
4
|
+
Attachment = Struct.new("Attachment", :file_name, :mime_type, :bytes, :date, :name, :is_upload, :headers, :url, keyword_init: true)
|
5
5
|
end
|
@@ -1,5 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module Trellodon
|
4
|
-
Card = Struct.new("Card", :id, :short_id, :name, :desc, :list_id, :labels, :comments, :attachments, :last_activity_date, keyword_init: true)
|
4
|
+
Card = Struct.new("Card", :id, :short_id, :name, :desc, :list_id, :labels, :comments, :attachments, :checklists, :last_activity_date, keyword_init: true)
|
5
5
|
end
|
data/lib/trellodon/entities.rb
CHANGED
@@ -5,3 +5,6 @@ require "trellodon/entities/card"
|
|
5
5
|
require "trellodon/entities/list"
|
6
6
|
require "trellodon/entities/comment"
|
7
7
|
require "trellodon/entities/attachment"
|
8
|
+
require "trellodon/entities/checklist"
|
9
|
+
require "trellodon/entities/checklist_item"
|
10
|
+
require "trellodon/entities/member"
|
@@ -0,0 +1,65 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Trellodon
|
4
|
+
class Executor
|
5
|
+
BOARD_ID_REGEX = /\/b\/([^\/]+)\//
|
6
|
+
private_constant :BOARD_ID_REGEX
|
7
|
+
|
8
|
+
def initialize(api_key:, api_token:, scheduler:, formatter:, logger: Config.logger)
|
9
|
+
@api_key = api_key
|
10
|
+
@api_token = api_token
|
11
|
+
@formatter = formatter
|
12
|
+
@logger = logger
|
13
|
+
@scheduler = scheduler
|
14
|
+
check_credentials!
|
15
|
+
|
16
|
+
@client = Client.new(api_key: @api_key, api_token: @api_token)
|
17
|
+
end
|
18
|
+
|
19
|
+
def download(board_pointer)
|
20
|
+
extract_board_id(board_pointer)
|
21
|
+
|
22
|
+
startup_time = Time.now
|
23
|
+
logger.debug "Fetching board 🚀️️"
|
24
|
+
board = scheduler.post do
|
25
|
+
client.fetch_board(board_id).tap { formatter.board_added(_1) }
|
26
|
+
end.value
|
27
|
+
|
28
|
+
logger.debug "Fetching #{board.card_ids.size} cards from the board with comments and attachments 🐢"
|
29
|
+
board.card_ids.map do |card_id|
|
30
|
+
scheduler.post do
|
31
|
+
client.fetch_card(card_id).tap { formatter.card_added(_1) }
|
32
|
+
end
|
33
|
+
end.map(&:value)
|
34
|
+
|
35
|
+
formatter.finish
|
36
|
+
logger.debug "All Trello API requests finished in #{(Time.now - startup_time).to_i} seconds ⌛"
|
37
|
+
end
|
38
|
+
|
39
|
+
private
|
40
|
+
|
41
|
+
attr_reader :client, :board_id, :logger, :scheduler, :formatter
|
42
|
+
|
43
|
+
def check_credentials!
|
44
|
+
return if @api_key.to_s.size.positive? && @api_token.to_s.size.positive?
|
45
|
+
|
46
|
+
raise ArgumentError, "Missing credentials. Please fill out both api_key, api_token first."
|
47
|
+
end
|
48
|
+
|
49
|
+
def extract_board_id(board_pointer)
|
50
|
+
@board_id = if URI::DEFAULT_PARSER.make_regexp.match?(board_pointer)
|
51
|
+
parse_board_url(board_pointer)
|
52
|
+
else
|
53
|
+
board_pointer
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def parse_board_url(board_url)
|
58
|
+
rel_path = URI.parse(board_url).request_uri
|
59
|
+
match_data = rel_path.match(BOARD_ID_REGEX)
|
60
|
+
raise ArgumentError, "Wrong trello board url" if match_data.nil? || match_data.size != 2
|
61
|
+
|
62
|
+
match_data[1]
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
@@ -9,13 +9,16 @@ module Trellodon
|
|
9
9
|
class Markdown < Base
|
10
10
|
attr_reader :output_dir
|
11
11
|
|
12
|
+
MD_FILENAME = "README.md"
|
13
|
+
ATTACHMENTS_DIRNAME = "attachments"
|
14
|
+
|
12
15
|
def initialize(output_dir:, **opts)
|
13
16
|
super(**opts)
|
14
17
|
@output_dir = output_dir
|
15
18
|
end
|
16
19
|
|
17
20
|
def card_added(card)
|
18
|
-
card_mdfile = File.join(card_path(card),
|
21
|
+
card_mdfile = File.join(card_path(card), MD_FILENAME)
|
19
22
|
raise "File #{card_mdfile} already exists" if File.exist?(card_mdfile)
|
20
23
|
|
21
24
|
File.write(card_mdfile, format_card(card))
|
@@ -29,7 +32,7 @@ module Trellodon
|
|
29
32
|
private
|
30
33
|
|
31
34
|
def card_path(card)
|
32
|
-
raise "Board is undefined"
|
35
|
+
raise "Board is undefined" unless @board
|
33
36
|
raise "List id is undefined" if card.list_id.nil? || card.list_id.empty?
|
34
37
|
list = @board.get_list(card.list_id)
|
35
38
|
raise "List #{card.list_id} is not found" if list.nil?
|
@@ -43,17 +46,16 @@ module Trellodon
|
|
43
46
|
end
|
44
47
|
|
45
48
|
def card_attachments_path(card)
|
46
|
-
attachments_dir = File.join(card_path(card),
|
49
|
+
attachments_dir = File.join(card_path(card), ATTACHMENTS_DIRNAME)
|
47
50
|
FileUtils.mkdir_p(attachments_dir) unless File.directory?(attachments_dir)
|
48
51
|
attachments_dir
|
49
52
|
end
|
50
53
|
|
51
54
|
def download_attachments(card)
|
52
|
-
return
|
53
|
-
return if card.attachments.nil? || card.attachments.empty? # rubocop:disable Lint/UnreachableCode
|
55
|
+
return if card.attachments.nil? || card.attachments.empty?
|
54
56
|
attachments_path = card_attachments_path(card)
|
55
|
-
card.attachments.each do |att|
|
56
|
-
IO.copy_stream(URI.parse(att.url).open, File.join(attachments_path, att.file_name))
|
57
|
+
card.attachments.select(&:is_upload).each do |att|
|
58
|
+
IO.copy_stream(URI.parse(att.url).open(att.headers), File.join(attachments_path, att.file_name))
|
57
59
|
end
|
58
60
|
end
|
59
61
|
|
@@ -61,7 +63,9 @@ module Trellodon
|
|
61
63
|
create_card_header(card) +
|
62
64
|
create_card_title(card) +
|
63
65
|
create_card_description(card) +
|
64
|
-
|
66
|
+
create_card_checklists(card) +
|
67
|
+
create_card_comments(card) +
|
68
|
+
create_card_attachments(card)
|
65
69
|
end
|
66
70
|
|
67
71
|
def create_card_header(card)
|
@@ -85,13 +89,35 @@ module Trellodon
|
|
85
89
|
<<~EOS
|
86
90
|
|
87
91
|
## Description
|
92
|
+
|
88
93
|
#{card.desc}
|
89
94
|
EOS
|
90
95
|
end
|
91
96
|
|
92
97
|
def create_card_labels(card)
|
93
98
|
return "" if card.labels.nil? || card.labels.empty?
|
94
|
-
card.labels.map { |label| "\n - " + label }.
|
99
|
+
card.labels.map { |label| "\n - " + label }.join
|
100
|
+
end
|
101
|
+
|
102
|
+
def create_card_checklists(card)
|
103
|
+
return "" if card.checklists.nil? || card.checklists.empty?
|
104
|
+
<<~EOS
|
105
|
+
|
106
|
+
## Checklists
|
107
|
+
#{card.checklists.map { |checklist| create_card_checklist(checklist) }.join}
|
108
|
+
EOS
|
109
|
+
end
|
110
|
+
|
111
|
+
def create_card_checklist(checklist)
|
112
|
+
<<~EOS
|
113
|
+
|
114
|
+
### #{checklist.name}
|
115
|
+
#{checklist.items&.map { |item| create_card_checklist_item(item) }&.join}
|
116
|
+
EOS
|
117
|
+
end
|
118
|
+
|
119
|
+
def create_card_checklist_item(item)
|
120
|
+
"\n- [#{item.checked? ? "x" : " "}] #{item.name}"
|
95
121
|
end
|
96
122
|
|
97
123
|
def create_card_comments(card)
|
@@ -99,17 +125,42 @@ module Trellodon
|
|
99
125
|
<<~EOS
|
100
126
|
|
101
127
|
## Comments
|
102
|
-
#{card.comments.map { |comment| create_card_comment(comment) }.
|
128
|
+
#{card.comments.map { |comment| create_card_comment(comment) }.join}
|
103
129
|
EOS
|
104
130
|
end
|
105
131
|
|
106
132
|
def create_card_comment(comment)
|
107
133
|
<<~EOS
|
108
134
|
|
109
|
-
**#{comment.
|
135
|
+
**#{comment.creator.full_name} (@#{comment.creator.username}) at #{comment.date}**
|
110
136
|
#{comment.text}
|
111
137
|
EOS
|
112
138
|
end
|
139
|
+
|
140
|
+
def create_card_attachments(card)
|
141
|
+
return "" if card.attachments.nil? || card.attachments.empty?
|
142
|
+
<<~EOS
|
143
|
+
|
144
|
+
## Attachments
|
145
|
+
#{card.attachments.map { |attachment| create_card_attachment(attachment) }.join}
|
146
|
+
EOS
|
147
|
+
end
|
148
|
+
|
149
|
+
def create_card_attachment(attachment)
|
150
|
+
<<~EOS
|
151
|
+
|
152
|
+
### #{attachment.name}
|
153
|
+
|
154
|
+
**date**: #{attachment.date}
|
155
|
+
**url**: #{attachment.url}
|
156
|
+
**local copy**: #{create_attachment_local_link(attachment)}
|
157
|
+
EOS
|
158
|
+
end
|
159
|
+
|
160
|
+
def create_attachment_local_link(attachment)
|
161
|
+
return "-" unless attachment.is_upload
|
162
|
+
"[#{attachment.file_name}](#{File.join("./", ATTACHMENTS_DIRNAME, attachment.file_name)})"
|
163
|
+
end
|
113
164
|
end
|
114
165
|
end
|
115
166
|
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trellodon/schedulers/base"
|
4
|
+
require "concurrent"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
module Schedulers
|
8
|
+
class Concurrent < Base
|
9
|
+
using RubyNext
|
10
|
+
|
11
|
+
DEFAULT_MAX_THREADS = 4
|
12
|
+
|
13
|
+
attr_reader :max_threads
|
14
|
+
|
15
|
+
def initialize(threads_count = DEFAULT_MAX_THREADS)
|
16
|
+
@max_threads = threads_count
|
17
|
+
end
|
18
|
+
|
19
|
+
def post(&block)
|
20
|
+
::Concurrent::Future.execute(executor:, &block)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def executor
|
26
|
+
::Concurrent::FixedThreadPool.new(max_threads)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/trellodon/schedulers.rb
CHANGED
data/lib/trellodon/version.rb
CHANGED
data/lib/trellodon.rb
CHANGED
@@ -1,8 +1,14 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "ruby-next"
|
4
|
+
|
5
|
+
require "ruby-next/language/setup"
|
6
|
+
RubyNext::Language.setup_gem_load_path(transpile: true)
|
7
|
+
|
3
8
|
require "trellodon/version"
|
4
9
|
require "trellodon/config"
|
5
|
-
require "trellodon/
|
10
|
+
require "trellodon/client"
|
11
|
+
require "trellodon/executor"
|
6
12
|
|
7
13
|
require "trellodon/entities"
|
8
14
|
require "trellodon/schedulers"
|
metadata
CHANGED
@@ -1,15 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: trellodon
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fargelus
|
8
8
|
- rinasergeeva
|
9
|
+
- palkan
|
9
10
|
autorequire:
|
10
11
|
bindir: bin
|
11
12
|
cert_chain: []
|
12
|
-
date: 2022-04-
|
13
|
+
date: 2022-04-12 00:00:00.000000000 Z
|
13
14
|
dependencies:
|
14
15
|
- !ruby/object:Gem::Dependency
|
15
16
|
name: anyway_config
|
@@ -53,12 +54,41 @@ dependencies:
|
|
53
54
|
- - ">="
|
54
55
|
- !ruby/object:Gem::Version
|
55
56
|
version: '0'
|
57
|
+
- !ruby/object:Gem::Dependency
|
58
|
+
name: concurrent-ruby
|
59
|
+
requirement: !ruby/object:Gem::Requirement
|
60
|
+
requirements:
|
61
|
+
- - ">="
|
62
|
+
- !ruby/object:Gem::Version
|
63
|
+
version: '0'
|
64
|
+
type: :runtime
|
65
|
+
prerelease: false
|
66
|
+
version_requirements: !ruby/object:Gem::Requirement
|
67
|
+
requirements:
|
68
|
+
- - ">="
|
69
|
+
- !ruby/object:Gem::Version
|
70
|
+
version: '0'
|
71
|
+
- !ruby/object:Gem::Dependency
|
72
|
+
name: ruby-next-core
|
73
|
+
requirement: !ruby/object:Gem::Requirement
|
74
|
+
requirements:
|
75
|
+
- - ">="
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: 0.15.0
|
78
|
+
type: :runtime
|
79
|
+
prerelease: false
|
80
|
+
version_requirements: !ruby/object:Gem::Requirement
|
81
|
+
requirements:
|
82
|
+
- - ">="
|
83
|
+
- !ruby/object:Gem::Version
|
84
|
+
version: 0.15.0
|
56
85
|
description: "\n The main purpose of Trellodon is to make it possible to backup
|
57
86
|
Trello boards to file system in a human-readable form (e.g., Markdown files).\n
|
58
87
|
\ "
|
59
88
|
email:
|
60
89
|
- ddraudred@gmail.com
|
61
90
|
- catherine.sergeeva@gmail.com
|
91
|
+
- dementiev.vm@gmail.com
|
62
92
|
executables:
|
63
93
|
- trellodon
|
64
94
|
extensions: []
|
@@ -70,22 +100,29 @@ files:
|
|
70
100
|
- bin/console
|
71
101
|
- bin/setup
|
72
102
|
- bin/trellodon
|
103
|
+
- lib/.rbnext/2.7/trellodon/executor.rb
|
104
|
+
- lib/.rbnext/3.0/trellodon/client.rb
|
105
|
+
- lib/.rbnext/3.1/trellodon/schedulers/concurrent.rb
|
73
106
|
- lib/trellodon.rb
|
74
|
-
- lib/trellodon/api_client.rb
|
75
|
-
- lib/trellodon/api_executor.rb
|
76
107
|
- lib/trellodon/cli.rb
|
108
|
+
- lib/trellodon/client.rb
|
77
109
|
- lib/trellodon/config.rb
|
78
110
|
- lib/trellodon/entities.rb
|
79
111
|
- lib/trellodon/entities/attachment.rb
|
80
112
|
- lib/trellodon/entities/board.rb
|
81
113
|
- lib/trellodon/entities/card.rb
|
114
|
+
- lib/trellodon/entities/checklist.rb
|
115
|
+
- lib/trellodon/entities/checklist_item.rb
|
82
116
|
- lib/trellodon/entities/comment.rb
|
83
117
|
- lib/trellodon/entities/list.rb
|
118
|
+
- lib/trellodon/entities/member.rb
|
119
|
+
- lib/trellodon/executor.rb
|
84
120
|
- lib/trellodon/formatters.rb
|
85
121
|
- lib/trellodon/formatters/base.rb
|
86
122
|
- lib/trellodon/formatters/markdown.rb
|
87
123
|
- lib/trellodon/schedulers.rb
|
88
124
|
- lib/trellodon/schedulers/base.rb
|
125
|
+
- lib/trellodon/schedulers/concurrent.rb
|
89
126
|
- lib/trellodon/schedulers/inline.rb
|
90
127
|
- lib/trellodon/schedulers/threaded.rb
|
91
128
|
- lib/trellodon/version.rb
|