trellodon 0.1.0 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +27 -1
- data/README.md +91 -2
- data/lib/{trellodon/api_executor.rb → .rbnext/2.7/trellodon/executor.rb} +6 -9
- data/lib/.rbnext/3.0/trellodon/client.rb +149 -0
- data/lib/.rbnext/3.1/trellodon/schedulers/concurrent.rb +30 -0
- data/lib/trellodon/cli.rb +68 -17
- data/lib/trellodon/{api_client.rb → client.rb} +53 -8
- data/lib/trellodon/entities/attachment.rb +1 -1
- data/lib/trellodon/entities/card.rb +1 -1
- data/lib/trellodon/entities/checklist.rb +5 -0
- data/lib/trellodon/entities/checklist_item.rb +9 -0
- data/lib/trellodon/entities/comment.rb +1 -1
- data/lib/trellodon/entities/member.rb +5 -0
- data/lib/trellodon/entities.rb +3 -0
- data/lib/trellodon/executor.rb +65 -0
- data/lib/trellodon/formatters/markdown.rb +62 -11
- data/lib/trellodon/schedulers/concurrent.rb +30 -0
- data/lib/trellodon/schedulers.rb +1 -0
- data/lib/trellodon/version.rb +1 -1
- data/lib/trellodon.rb +7 -1
- metadata +41 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a61911e92509a77f13aa90f9f56c14d038dbe443b4e1063d28f66b6745b04432
|
4
|
+
data.tar.gz: '079609a773107e786a7d795fe7d23564ded2aa66e8d587a6fd22b4e288e15478'
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 617119b48c1f2385c87d57ffd2683933097ed1aae11a51577f65e16012ab1f58b65187f57884123ed04ba522797925826e3cf202e7d4dad9d1fd63a3d736bcef
|
7
|
+
data.tar.gz: 30c1f1bcb2df2c69a3c8d0cebc93a597f16d2207d33bef979670134d306bd0173cc97948f38f739cf1d6b5eef0a7a602112fb0de40021d078d0188280ca4d039
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,31 @@
|
|
1
|
-
|
1
|
+
# Change log
|
2
|
+
|
3
|
+
## [master]
|
4
|
+
|
5
|
+
## [0.3.0] - 2022-04-12 🚀🚀🚀
|
6
|
+
|
7
|
+
- Handle missing members. ([@palkan][])
|
8
|
+
|
9
|
+
- Use inline scheduler by default. ([@palkan][])
|
10
|
+
|
11
|
+
- Replace `--sync` with `--concurrency=0`. ([@palkan][])
|
12
|
+
|
13
|
+
## [0.2.1] - 2022-04-12 🚀
|
14
|
+
|
15
|
+
- Misc changes.
|
16
|
+
|
17
|
+
## [0.2.0] - 2022-04-12
|
18
|
+
|
19
|
+
- Store Trello API creds locally for avoid asking it before execution ([@fargelus][]).
|
20
|
+
- Download attachments, human names in comments, rename index.md to README.md ([@rinasergeeva][]).
|
21
|
+
- Added minimal project README.md ([@rinasergeeva][], [@fargelus][]).
|
22
|
+
- --sync param for switch to `Schedulers::Inline` in CLI ([@fargelus][]).
|
23
|
+
- Concurrent scheduler with customizable thread pool size ([@fargelus][]).
|
24
|
+
- Checklists support ([@rinasergeeva][]).
|
2
25
|
|
3
26
|
## [0.1.0] - 2022-04-07
|
4
27
|
|
5
28
|
- Initial release
|
29
|
+
|
30
|
+
[@fargelus]: https://github.com/fargelus
|
31
|
+
[@rinasergeeva]: https://github.com/rinasergeeva
|
data/README.md
CHANGED
@@ -1,3 +1,92 @@
|
|
1
|
-
|
1
|
+
![Build](https://github.com/evilmartians/trellodon/workflows/Build/badge.svg)
|
2
|
+
[![Gem Version](https://badge.fury.io/rb/trellodon.svg)](https://rubygems.org/gems/trellodon)
|
2
3
|
|
3
|
-
|
4
|
+
# Trellodon
|
5
|
+
|
6
|
+
A Ruby tool to export a Trello board and convert it into a set of folders and markdown files, corresponding to lists and cards on the board. For each card, you’ll get the basic details such as:
|
7
|
+
|
8
|
+
* Name
|
9
|
+
* Description
|
10
|
+
* Labels
|
11
|
+
* Comments
|
12
|
+
* Attachments
|
13
|
+
|
14
|
+
<img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54"></a>
|
15
|
+
|
16
|
+
## Installation
|
17
|
+
|
18
|
+
```
|
19
|
+
gem install trellodon
|
20
|
+
```
|
21
|
+
|
22
|
+
## Prerequisites
|
23
|
+
|
24
|
+
Trellodon needs two secret codes to download your boards:
|
25
|
+
|
26
|
+
* Trello API key
|
27
|
+
* Authentication token
|
28
|
+
|
29
|
+
To generate them, go to the [trello.com/app-key](https://trello.com/app-key) page and follow the instructions.
|
30
|
+
|
31
|
+
## Usage
|
32
|
+
|
33
|
+
```sh
|
34
|
+
Usage: trellodon dump [options]
|
35
|
+
Options:
|
36
|
+
--board VALUE Board URL or ID
|
37
|
+
--out VALUE Destination folder path
|
38
|
+
--concurrency VALUE Amount of processing threads (default: 4). Set to 0 to execute API requests in-line
|
39
|
+
--clear-auth Remove saved api credentials
|
40
|
+
-v, --version Print current version
|
41
|
+
-h, --help Print help
|
42
|
+
```
|
43
|
+
|
44
|
+
### Detailed example
|
45
|
+
|
46
|
+
Suppose we have a Trello board called `projects` with lists `Brainstorm`, `TODO`, `DOING`, `DONE`. Lists contains own cards.<br>
|
47
|
+
All Trello boards have its own id this id requires Trellodon to fetch board via Trello API.
|
48
|
+
You can find this id in two ways:
|
49
|
+
|
50
|
+
* In URL `https://trello.com/b/{id}/projects` your Trello board id will be placed as `{id}` in example;
|
51
|
+
* Put `.json` in your board URL like this `https://trello.com/b/{id}/projects.json` id field in json output is what you need;
|
52
|
+
|
53
|
+
You can also paste whole board URL and Trellodon will parse it correctly.<br>
|
54
|
+
After launch Trellodon creates follow output in specified folder:
|
55
|
+
|
56
|
+
```sh
|
57
|
+
projects/
|
58
|
+
Brainstorm/
|
59
|
+
first_card_title/
|
60
|
+
README.md
|
61
|
+
attachments/
|
62
|
+
image.png
|
63
|
+
TODO/
|
64
|
+
first_card_title/
|
65
|
+
README.md
|
66
|
+
attachments/
|
67
|
+
report.docx
|
68
|
+
...
|
69
|
+
```
|
70
|
+
|
71
|
+
Each card has follow output format:
|
72
|
+
|
73
|
+
```md
|
74
|
+
---
|
75
|
+
title: card_title
|
76
|
+
last_updated_at: 2022-03-16 16:28:39 UTC
|
77
|
+
labels: Test
|
78
|
+
---
|
79
|
+
|
80
|
+
# card_title
|
81
|
+
|
82
|
+
## Description
|
83
|
+
Some card description
|
84
|
+
|
85
|
+
## Comments
|
86
|
+
** John Doe @doe at 2022-03-16 16:28:39 UTC**
|
87
|
+
|
88
|
+
## Attachments
|
89
|
+
### Image.png
|
90
|
+
**date**: 2022-03-16
|
91
|
+
**url**: https://trello.com/1/cards/#{card_id}/attachments/#{attachment.id}/download/image.png
|
92
|
+
```
|
@@ -1,10 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require "trellodon/api_client"
|
4
|
-
require "trellodon/config"
|
5
|
-
|
6
3
|
module Trellodon
|
7
|
-
class
|
4
|
+
class Executor
|
8
5
|
BOARD_ID_REGEX = /\/b\/([^\/]+)\//
|
9
6
|
private_constant :BOARD_ID_REGEX
|
10
7
|
|
@@ -16,7 +13,7 @@ module Trellodon
|
|
16
13
|
@scheduler = scheduler
|
17
14
|
check_credentials!
|
18
15
|
|
19
|
-
@
|
16
|
+
@client = Client.new(api_key: @api_key, api_token: @api_token)
|
20
17
|
end
|
21
18
|
|
22
19
|
def download(board_pointer)
|
@@ -25,13 +22,13 @@ module Trellodon
|
|
25
22
|
startup_time = Time.now
|
26
23
|
logger.debug "Fetching board 🚀️️"
|
27
24
|
board = scheduler.post do
|
28
|
-
|
25
|
+
client.fetch_board(board_id).tap { |_1| formatter.board_added(_1) }
|
29
26
|
end.value
|
30
27
|
|
31
|
-
logger.debug "Fetching cards
|
28
|
+
logger.debug "Fetching #{board.card_ids.size} cards from the board with comments and attachments 🐢"
|
32
29
|
board.card_ids.map do |card_id|
|
33
30
|
scheduler.post do
|
34
|
-
|
31
|
+
client.fetch_card(card_id).tap { |_1| formatter.card_added(_1) }
|
35
32
|
end
|
36
33
|
end.map(&:value)
|
37
34
|
|
@@ -41,7 +38,7 @@ module Trellodon
|
|
41
38
|
|
42
39
|
private
|
43
40
|
|
44
|
-
attr_reader :
|
41
|
+
attr_reader :client, :board_id, :logger, :scheduler, :formatter
|
45
42
|
|
46
43
|
def check_credentials!
|
47
44
|
return if @api_key.to_s.size.positive? && @api_token.to_s.size.positive?
|
@@ -0,0 +1,149 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trello"
|
4
|
+
require "trellodon/entities"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
class Client
|
8
|
+
class DeletedMember < Struct.new(:id)
|
9
|
+
def full_name ; "[DELETED]"; end
|
10
|
+
|
11
|
+
def username ; "__deleted__"; end
|
12
|
+
end
|
13
|
+
|
14
|
+
def initialize(api_key:, api_token:, logger: Config.logger)
|
15
|
+
@logger = logger
|
16
|
+
@client = Trello::Client.new(
|
17
|
+
developer_public_key: api_key,
|
18
|
+
member_token: api_token
|
19
|
+
)
|
20
|
+
@members = {}
|
21
|
+
end
|
22
|
+
|
23
|
+
def fetch_board(board_id)
|
24
|
+
retrying do
|
25
|
+
board = @client.find(:boards, board_id)
|
26
|
+
|
27
|
+
Board.new(
|
28
|
+
id: board.id,
|
29
|
+
short_id: board.short_url.chars.last(8).join,
|
30
|
+
name: board.name,
|
31
|
+
lists: lists_from(board),
|
32
|
+
card_ids: @client.find_many(Trello::Card, "/boards/#{board_id}/cards", fields: "id").map(&:id),
|
33
|
+
last_activity_date: board.last_activity_date
|
34
|
+
)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
def fetch_card(card_id)
|
39
|
+
retrying do
|
40
|
+
card = @client.find(:cards, card_id)
|
41
|
+
|
42
|
+
Card.new(
|
43
|
+
id: card.id,
|
44
|
+
short_id: card.short_url.chars.last(8).join,
|
45
|
+
name: card.name,
|
46
|
+
desc: card.desc,
|
47
|
+
labels: card.labels.map(&:name),
|
48
|
+
list_id: card.list_id,
|
49
|
+
comments: comments_from(card),
|
50
|
+
attachments: attachments_from(card),
|
51
|
+
checklists: checklists_from(card),
|
52
|
+
last_activity_date: card.last_activity_date
|
53
|
+
)
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
private
|
58
|
+
|
59
|
+
attr_reader :logger
|
60
|
+
|
61
|
+
def retrying(&block)
|
62
|
+
attempt = 0
|
63
|
+
begin
|
64
|
+
block.call
|
65
|
+
rescue Trello::Error => err
|
66
|
+
raise unless err.status_code == 429
|
67
|
+
attempt += 1
|
68
|
+
cooldown = 2**attempt + rand(2**attempt) - 2**(attempt - 1)
|
69
|
+
logger.warn "API limit exceeded, cool down for #{cooldown}s"
|
70
|
+
sleep cooldown
|
71
|
+
retry
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
def comments_from(card)
|
76
|
+
card.comments.map do |comment|
|
77
|
+
Comment.new(
|
78
|
+
text: comment.data["text"],
|
79
|
+
date: comment.date,
|
80
|
+
creator: member_from(comment)
|
81
|
+
)
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
def attachments_from(card)
|
86
|
+
card.attachments.map do |attach|
|
87
|
+
Attachment.new(
|
88
|
+
file_name: attach.file_name,
|
89
|
+
mime_type: attach.mime_type,
|
90
|
+
bytes: attach.bytes,
|
91
|
+
date: attach.date,
|
92
|
+
name: attach.name,
|
93
|
+
is_upload: attach.is_upload,
|
94
|
+
headers: attach.is_upload ? {"Authorization" => "OAuth oauth_consumer_key=\"#{@client.configuration.developer_public_key}\", oauth_token=\"#{@client.configuration.member_token}\""} : {},
|
95
|
+
url: attach.url
|
96
|
+
)
|
97
|
+
end
|
98
|
+
end
|
99
|
+
|
100
|
+
def lists_from(board)
|
101
|
+
board.lists.map do |list|
|
102
|
+
List.new(
|
103
|
+
id: list.id,
|
104
|
+
name: list.name,
|
105
|
+
short_id: board.short_url.chars.last(8).join
|
106
|
+
)
|
107
|
+
end
|
108
|
+
end
|
109
|
+
|
110
|
+
def checklists_from(card)
|
111
|
+
card.checklists.map do |checklist|
|
112
|
+
Checklist.new(
|
113
|
+
id: checklist.id,
|
114
|
+
name: checklist.name,
|
115
|
+
items: checklist_items_from(checklist)
|
116
|
+
)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
def checklist_items_from(checklist)
|
121
|
+
checklist.items.map do |item|
|
122
|
+
ChecklistItem.new(
|
123
|
+
id: item.id,
|
124
|
+
name: item.name,
|
125
|
+
state: item.state
|
126
|
+
)
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
def member_from(comment)
|
131
|
+
return @members[comment.creator_id] if @members.has_key? comment.creator_id
|
132
|
+
|
133
|
+
member =
|
134
|
+
begin
|
135
|
+
@client.find(:members, comment.creator_id)
|
136
|
+
rescue Trello::Error => err
|
137
|
+
raise unless err.status_code == 404
|
138
|
+
|
139
|
+
DeletedMember.new(comment.creator_id)
|
140
|
+
end
|
141
|
+
|
142
|
+
@members[comment.creator_id] = Member.new(
|
143
|
+
id: member.id,
|
144
|
+
full_name: member.full_name,
|
145
|
+
username: member.username
|
146
|
+
)
|
147
|
+
end
|
148
|
+
end
|
149
|
+
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trellodon/schedulers/base"
|
4
|
+
require "concurrent"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
module Schedulers
|
8
|
+
class Concurrent < Base
|
9
|
+
using RubyNext
|
10
|
+
|
11
|
+
DEFAULT_MAX_THREADS = 4
|
12
|
+
|
13
|
+
attr_reader :max_threads
|
14
|
+
|
15
|
+
def initialize(threads_count = DEFAULT_MAX_THREADS)
|
16
|
+
@max_threads = threads_count
|
17
|
+
end
|
18
|
+
|
19
|
+
def post(&block)
|
20
|
+
::Concurrent::Future.execute(executor: executor, &block)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def executor
|
26
|
+
::Concurrent::FixedThreadPool.new(max_threads)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/trellodon/cli.rb
CHANGED
@@ -1,36 +1,34 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "trellodon"
|
4
|
-
|
5
4
|
require "tty-prompt"
|
5
|
+
require "fileutils"
|
6
6
|
|
7
7
|
module Trellodon
|
8
8
|
class CLI
|
9
|
-
class Prompt
|
10
|
-
|
11
|
-
@prompt = TTY::Prompt.new
|
12
|
-
end
|
9
|
+
class Prompt < TTY::Prompt
|
10
|
+
include Singleton
|
13
11
|
|
14
12
|
def ask_api_key
|
15
|
-
|
13
|
+
mask(
|
16
14
|
"Provide your Developer API Key (see https://trello.com/app-key):"
|
17
15
|
) { |q| q.required true }
|
18
16
|
end
|
19
17
|
|
20
18
|
def ask_api_token
|
21
|
-
|
19
|
+
mask(
|
22
20
|
"Provide your API token:"
|
23
21
|
) { |q| q.required true }
|
24
22
|
end
|
25
23
|
|
26
24
|
def ask_board
|
27
|
-
|
25
|
+
ask("Which board would you like to dump? (URL or ID)") do |q|
|
28
26
|
q.required true
|
29
27
|
end
|
30
28
|
end
|
31
29
|
|
32
30
|
def ask_folder
|
33
|
-
|
31
|
+
ask("Destination folder?", default: "./")
|
34
32
|
end
|
35
33
|
end
|
36
34
|
|
@@ -44,10 +42,15 @@ module Trellodon
|
|
44
42
|
out: "Destination folder path"
|
45
43
|
)
|
46
44
|
|
47
|
-
extend_options do |opts,
|
48
|
-
opts.banner = "Usage: trellodon dump\n"\
|
45
|
+
extend_options do |opts, config|
|
46
|
+
opts.banner = "Usage: trellodon dump \e[34m[options]\e[0m\n"\
|
49
47
|
"Options:\n"
|
50
48
|
|
49
|
+
opts.on("--concurrency VALUE", "Amount of processing threads (default: 4)")
|
50
|
+
opts.on("--clear-auth", "Remove saved api credentials") do
|
51
|
+
config.reload(api_token: nil, api_key: nil)
|
52
|
+
end
|
53
|
+
|
51
54
|
opts.on_tail("-v", "--version", "Print current version") do
|
52
55
|
puts Trellodon::VERSION
|
53
56
|
exit
|
@@ -59,8 +62,25 @@ module Trellodon
|
|
59
62
|
end
|
60
63
|
end
|
61
64
|
|
65
|
+
def resolve_config_path(_, _)
|
66
|
+
config_path
|
67
|
+
end
|
68
|
+
|
69
|
+
def config_path
|
70
|
+
File.join(Dir.home, ".config", "#{config_name}.yml")
|
71
|
+
end
|
72
|
+
|
73
|
+
def update_config_file!
|
74
|
+
dir = File.dirname(config_path)
|
75
|
+
FileUtils.mkdir_p(dir) unless File.exist?(dir)
|
76
|
+
|
77
|
+
yml = "api_token: #{api_token}\n"\
|
78
|
+
"api_key: #{api_key}"
|
79
|
+
File.write(config_path, yml)
|
80
|
+
end
|
81
|
+
|
62
82
|
private def prompt
|
63
|
-
|
83
|
+
Prompt.instance
|
64
84
|
end
|
65
85
|
|
66
86
|
def api_key
|
@@ -82,34 +102,65 @@ module Trellodon
|
|
82
102
|
|
83
103
|
def initialize
|
84
104
|
@config = Config.new
|
105
|
+
@scheduler = Schedulers::Inline.new
|
85
106
|
end
|
86
107
|
|
87
108
|
def run
|
88
109
|
check_command!
|
89
110
|
|
90
|
-
|
111
|
+
need_to_fill_config = config.to_h[:api_key].nil?
|
112
|
+
|
113
|
+
executor = Trellodon::Executor.new(
|
91
114
|
api_key: config.api_key,
|
92
115
|
api_token: config.api_token,
|
93
116
|
formatter: Trellodon::Formatters::Markdown.new(output_dir: config.out),
|
94
|
-
scheduler:
|
117
|
+
scheduler: scheduler
|
95
118
|
)
|
119
|
+
|
120
|
+
ask_save_credentials if need_to_fill_config
|
121
|
+
|
96
122
|
executor.download(config.board)
|
97
123
|
end
|
98
124
|
|
99
125
|
private
|
100
126
|
|
101
|
-
attr_reader :config
|
127
|
+
attr_reader :config, :scheduler
|
128
|
+
|
129
|
+
def prompt
|
130
|
+
Prompt.instance
|
131
|
+
end
|
102
132
|
|
103
133
|
def check_command!
|
104
|
-
|
134
|
+
params = {}
|
135
|
+
command, = config.option_parser.parse!(ARGV, into: params)
|
105
136
|
unless command
|
106
137
|
puts config.option_parser.help
|
107
138
|
exit
|
108
139
|
end
|
109
140
|
|
110
|
-
|
141
|
+
if command == "dump"
|
142
|
+
change_scheduler(params) if params.key?(:sync) || params.key?(:concurrency)
|
143
|
+
return
|
144
|
+
end
|
111
145
|
|
112
146
|
raise "Unknown command: #{command}. Available commands: dump"
|
113
147
|
end
|
148
|
+
|
149
|
+
def ask_save_credentials
|
150
|
+
answer = prompt.yes?("Would you like to save api creds so next time trellodon won't asking it?", default: true)
|
151
|
+
config.update_config_file! if answer
|
152
|
+
end
|
153
|
+
|
154
|
+
def change_scheduler(params)
|
155
|
+
return unless params[:concurrency]
|
156
|
+
|
157
|
+
threads = Integer(params[:concurrency])
|
158
|
+
|
159
|
+
return @scheduler = Schedulers::Inline.new if threads.zero?
|
160
|
+
|
161
|
+
raise "Value of concurrency option must be integer and greater than zero" unless threads.positive?
|
162
|
+
|
163
|
+
@scheduler = Schedulers::Concurrent.new(threads)
|
164
|
+
end
|
114
165
|
end
|
115
166
|
end
|
@@ -1,17 +1,23 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "trello"
|
4
|
-
|
5
4
|
require "trellodon/entities"
|
6
5
|
|
7
6
|
module Trellodon
|
8
|
-
class
|
7
|
+
class Client
|
8
|
+
class DeletedMember < Struct.new(:id)
|
9
|
+
def full_name = "[DELETED]"
|
10
|
+
|
11
|
+
def username = "__deleted__"
|
12
|
+
end
|
13
|
+
|
9
14
|
def initialize(api_key:, api_token:, logger: Config.logger)
|
10
15
|
@logger = logger
|
11
16
|
@client = Trello::Client.new(
|
12
17
|
developer_public_key: api_key,
|
13
18
|
member_token: api_token
|
14
19
|
)
|
20
|
+
@members = {}
|
15
21
|
end
|
16
22
|
|
17
23
|
def fetch_board(board_id)
|
@@ -42,6 +48,7 @@ module Trellodon
|
|
42
48
|
list_id: card.list_id,
|
43
49
|
comments: comments_from(card),
|
44
50
|
attachments: attachments_from(card),
|
51
|
+
checklists: checklists_from(card),
|
45
52
|
last_activity_date: card.last_activity_date
|
46
53
|
)
|
47
54
|
end
|
@@ -57,15 +64,10 @@ module Trellodon
|
|
57
64
|
block.call
|
58
65
|
rescue Trello::Error => err
|
59
66
|
raise unless err.status_code == 429
|
60
|
-
|
61
67
|
attempt += 1
|
62
|
-
|
63
68
|
cooldown = 2**attempt + rand(2**attempt) - 2**(attempt - 1)
|
64
|
-
|
65
69
|
logger.warn "API limit exceeded, cool down for #{cooldown}s"
|
66
|
-
|
67
70
|
sleep cooldown
|
68
|
-
|
69
71
|
retry
|
70
72
|
end
|
71
73
|
end
|
@@ -75,7 +77,7 @@ module Trellodon
|
|
75
77
|
Comment.new(
|
76
78
|
text: comment.data["text"],
|
77
79
|
date: comment.date,
|
78
|
-
|
80
|
+
creator: member_from(comment)
|
79
81
|
)
|
80
82
|
end
|
81
83
|
end
|
@@ -86,6 +88,10 @@ module Trellodon
|
|
86
88
|
file_name: attach.file_name,
|
87
89
|
mime_type: attach.mime_type,
|
88
90
|
bytes: attach.bytes,
|
91
|
+
date: attach.date,
|
92
|
+
name: attach.name,
|
93
|
+
is_upload: attach.is_upload,
|
94
|
+
headers: attach.is_upload ? {"Authorization" => "OAuth oauth_consumer_key=\"#{@client.configuration.developer_public_key}\", oauth_token=\"#{@client.configuration.member_token}\""} : {},
|
89
95
|
url: attach.url
|
90
96
|
)
|
91
97
|
end
|
@@ -100,5 +106,44 @@ module Trellodon
|
|
100
106
|
)
|
101
107
|
end
|
102
108
|
end
|
109
|
+
|
110
|
+
def checklists_from(card)
|
111
|
+
card.checklists.map do |checklist|
|
112
|
+
Checklist.new(
|
113
|
+
id: checklist.id,
|
114
|
+
name: checklist.name,
|
115
|
+
items: checklist_items_from(checklist)
|
116
|
+
)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
def checklist_items_from(checklist)
|
121
|
+
checklist.items.map do |item|
|
122
|
+
ChecklistItem.new(
|
123
|
+
id: item.id,
|
124
|
+
name: item.name,
|
125
|
+
state: item.state
|
126
|
+
)
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
def member_from(comment)
|
131
|
+
return @members[comment.creator_id] if @members.has_key? comment.creator_id
|
132
|
+
|
133
|
+
member =
|
134
|
+
begin
|
135
|
+
@client.find(:members, comment.creator_id)
|
136
|
+
rescue Trello::Error => err
|
137
|
+
raise unless err.status_code == 404
|
138
|
+
|
139
|
+
DeletedMember.new(comment.creator_id)
|
140
|
+
end
|
141
|
+
|
142
|
+
@members[comment.creator_id] = Member.new(
|
143
|
+
id: member.id,
|
144
|
+
full_name: member.full_name,
|
145
|
+
username: member.username
|
146
|
+
)
|
147
|
+
end
|
103
148
|
end
|
104
149
|
end
|
@@ -1,5 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module Trellodon
|
4
|
-
Attachment = Struct.new("Attachment", :file_name, :mime_type, :bytes, :url, keyword_init: true)
|
4
|
+
Attachment = Struct.new("Attachment", :file_name, :mime_type, :bytes, :date, :name, :is_upload, :headers, :url, keyword_init: true)
|
5
5
|
end
|
@@ -1,5 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module Trellodon
|
4
|
-
Card = Struct.new("Card", :id, :short_id, :name, :desc, :list_id, :labels, :comments, :attachments, :last_activity_date, keyword_init: true)
|
4
|
+
Card = Struct.new("Card", :id, :short_id, :name, :desc, :list_id, :labels, :comments, :attachments, :checklists, :last_activity_date, keyword_init: true)
|
5
5
|
end
|
data/lib/trellodon/entities.rb
CHANGED
@@ -5,3 +5,6 @@ require "trellodon/entities/card"
|
|
5
5
|
require "trellodon/entities/list"
|
6
6
|
require "trellodon/entities/comment"
|
7
7
|
require "trellodon/entities/attachment"
|
8
|
+
require "trellodon/entities/checklist"
|
9
|
+
require "trellodon/entities/checklist_item"
|
10
|
+
require "trellodon/entities/member"
|
@@ -0,0 +1,65 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Trellodon
|
4
|
+
class Executor
|
5
|
+
BOARD_ID_REGEX = /\/b\/([^\/]+)\//
|
6
|
+
private_constant :BOARD_ID_REGEX
|
7
|
+
|
8
|
+
def initialize(api_key:, api_token:, scheduler:, formatter:, logger: Config.logger)
|
9
|
+
@api_key = api_key
|
10
|
+
@api_token = api_token
|
11
|
+
@formatter = formatter
|
12
|
+
@logger = logger
|
13
|
+
@scheduler = scheduler
|
14
|
+
check_credentials!
|
15
|
+
|
16
|
+
@client = Client.new(api_key: @api_key, api_token: @api_token)
|
17
|
+
end
|
18
|
+
|
19
|
+
def download(board_pointer)
|
20
|
+
extract_board_id(board_pointer)
|
21
|
+
|
22
|
+
startup_time = Time.now
|
23
|
+
logger.debug "Fetching board 🚀️️"
|
24
|
+
board = scheduler.post do
|
25
|
+
client.fetch_board(board_id).tap { formatter.board_added(_1) }
|
26
|
+
end.value
|
27
|
+
|
28
|
+
logger.debug "Fetching #{board.card_ids.size} cards from the board with comments and attachments 🐢"
|
29
|
+
board.card_ids.map do |card_id|
|
30
|
+
scheduler.post do
|
31
|
+
client.fetch_card(card_id).tap { formatter.card_added(_1) }
|
32
|
+
end
|
33
|
+
end.map(&:value)
|
34
|
+
|
35
|
+
formatter.finish
|
36
|
+
logger.debug "All Trello API requests finished in #{(Time.now - startup_time).to_i} seconds ⌛"
|
37
|
+
end
|
38
|
+
|
39
|
+
private
|
40
|
+
|
41
|
+
attr_reader :client, :board_id, :logger, :scheduler, :formatter
|
42
|
+
|
43
|
+
def check_credentials!
|
44
|
+
return if @api_key.to_s.size.positive? && @api_token.to_s.size.positive?
|
45
|
+
|
46
|
+
raise ArgumentError, "Missing credentials. Please fill out both api_key, api_token first."
|
47
|
+
end
|
48
|
+
|
49
|
+
def extract_board_id(board_pointer)
|
50
|
+
@board_id = if URI::DEFAULT_PARSER.make_regexp.match?(board_pointer)
|
51
|
+
parse_board_url(board_pointer)
|
52
|
+
else
|
53
|
+
board_pointer
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def parse_board_url(board_url)
|
58
|
+
rel_path = URI.parse(board_url).request_uri
|
59
|
+
match_data = rel_path.match(BOARD_ID_REGEX)
|
60
|
+
raise ArgumentError, "Wrong trello board url" if match_data.nil? || match_data.size != 2
|
61
|
+
|
62
|
+
match_data[1]
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
@@ -9,13 +9,16 @@ module Trellodon
|
|
9
9
|
class Markdown < Base
|
10
10
|
attr_reader :output_dir
|
11
11
|
|
12
|
+
MD_FILENAME = "README.md"
|
13
|
+
ATTACHMENTS_DIRNAME = "attachments"
|
14
|
+
|
12
15
|
def initialize(output_dir:, **opts)
|
13
16
|
super(**opts)
|
14
17
|
@output_dir = output_dir
|
15
18
|
end
|
16
19
|
|
17
20
|
def card_added(card)
|
18
|
-
card_mdfile = File.join(card_path(card),
|
21
|
+
card_mdfile = File.join(card_path(card), MD_FILENAME)
|
19
22
|
raise "File #{card_mdfile} already exists" if File.exist?(card_mdfile)
|
20
23
|
|
21
24
|
File.write(card_mdfile, format_card(card))
|
@@ -29,7 +32,7 @@ module Trellodon
|
|
29
32
|
private
|
30
33
|
|
31
34
|
def card_path(card)
|
32
|
-
raise "Board is undefined"
|
35
|
+
raise "Board is undefined" unless @board
|
33
36
|
raise "List id is undefined" if card.list_id.nil? || card.list_id.empty?
|
34
37
|
list = @board.get_list(card.list_id)
|
35
38
|
raise "List #{card.list_id} is not found" if list.nil?
|
@@ -43,17 +46,16 @@ module Trellodon
|
|
43
46
|
end
|
44
47
|
|
45
48
|
def card_attachments_path(card)
|
46
|
-
attachments_dir = File.join(card_path(card),
|
49
|
+
attachments_dir = File.join(card_path(card), ATTACHMENTS_DIRNAME)
|
47
50
|
FileUtils.mkdir_p(attachments_dir) unless File.directory?(attachments_dir)
|
48
51
|
attachments_dir
|
49
52
|
end
|
50
53
|
|
51
54
|
def download_attachments(card)
|
52
|
-
return
|
53
|
-
return if card.attachments.nil? || card.attachments.empty? # rubocop:disable Lint/UnreachableCode
|
55
|
+
return if card.attachments.nil? || card.attachments.empty?
|
54
56
|
attachments_path = card_attachments_path(card)
|
55
|
-
card.attachments.each do |att|
|
56
|
-
IO.copy_stream(URI.parse(att.url).open, File.join(attachments_path, att.file_name))
|
57
|
+
card.attachments.select(&:is_upload).each do |att|
|
58
|
+
IO.copy_stream(URI.parse(att.url).open(att.headers), File.join(attachments_path, att.file_name))
|
57
59
|
end
|
58
60
|
end
|
59
61
|
|
@@ -61,7 +63,9 @@ module Trellodon
|
|
61
63
|
create_card_header(card) +
|
62
64
|
create_card_title(card) +
|
63
65
|
create_card_description(card) +
|
64
|
-
|
66
|
+
create_card_checklists(card) +
|
67
|
+
create_card_comments(card) +
|
68
|
+
create_card_attachments(card)
|
65
69
|
end
|
66
70
|
|
67
71
|
def create_card_header(card)
|
@@ -85,13 +89,35 @@ module Trellodon
|
|
85
89
|
<<~EOS
|
86
90
|
|
87
91
|
## Description
|
92
|
+
|
88
93
|
#{card.desc}
|
89
94
|
EOS
|
90
95
|
end
|
91
96
|
|
92
97
|
def create_card_labels(card)
|
93
98
|
return "" if card.labels.nil? || card.labels.empty?
|
94
|
-
card.labels.map { |label| "\n - " + label }.
|
99
|
+
card.labels.map { |label| "\n - " + label }.join
|
100
|
+
end
|
101
|
+
|
102
|
+
def create_card_checklists(card)
|
103
|
+
return "" if card.checklists.nil? || card.checklists.empty?
|
104
|
+
<<~EOS
|
105
|
+
|
106
|
+
## Checklists
|
107
|
+
#{card.checklists.map { |checklist| create_card_checklist(checklist) }.join}
|
108
|
+
EOS
|
109
|
+
end
|
110
|
+
|
111
|
+
def create_card_checklist(checklist)
|
112
|
+
<<~EOS
|
113
|
+
|
114
|
+
### #{checklist.name}
|
115
|
+
#{checklist.items&.map { |item| create_card_checklist_item(item) }&.join}
|
116
|
+
EOS
|
117
|
+
end
|
118
|
+
|
119
|
+
def create_card_checklist_item(item)
|
120
|
+
"\n- [#{item.checked? ? "x" : " "}] #{item.name}"
|
95
121
|
end
|
96
122
|
|
97
123
|
def create_card_comments(card)
|
@@ -99,17 +125,42 @@ module Trellodon
|
|
99
125
|
<<~EOS
|
100
126
|
|
101
127
|
## Comments
|
102
|
-
#{card.comments.map { |comment| create_card_comment(comment) }.
|
128
|
+
#{card.comments.map { |comment| create_card_comment(comment) }.join}
|
103
129
|
EOS
|
104
130
|
end
|
105
131
|
|
106
132
|
def create_card_comment(comment)
|
107
133
|
<<~EOS
|
108
134
|
|
109
|
-
**#{comment.
|
135
|
+
**#{comment.creator.full_name} (@#{comment.creator.username}) at #{comment.date}**
|
110
136
|
#{comment.text}
|
111
137
|
EOS
|
112
138
|
end
|
139
|
+
|
140
|
+
def create_card_attachments(card)
|
141
|
+
return "" if card.attachments.nil? || card.attachments.empty?
|
142
|
+
<<~EOS
|
143
|
+
|
144
|
+
## Attachments
|
145
|
+
#{card.attachments.map { |attachment| create_card_attachment(attachment) }.join}
|
146
|
+
EOS
|
147
|
+
end
|
148
|
+
|
149
|
+
def create_card_attachment(attachment)
|
150
|
+
<<~EOS
|
151
|
+
|
152
|
+
### #{attachment.name}
|
153
|
+
|
154
|
+
**date**: #{attachment.date}
|
155
|
+
**url**: #{attachment.url}
|
156
|
+
**local copy**: #{create_attachment_local_link(attachment)}
|
157
|
+
EOS
|
158
|
+
end
|
159
|
+
|
160
|
+
def create_attachment_local_link(attachment)
|
161
|
+
return "-" unless attachment.is_upload
|
162
|
+
"[#{attachment.file_name}](#{File.join("./", ATTACHMENTS_DIRNAME, attachment.file_name)})"
|
163
|
+
end
|
113
164
|
end
|
114
165
|
end
|
115
166
|
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "trellodon/schedulers/base"
|
4
|
+
require "concurrent"
|
5
|
+
|
6
|
+
module Trellodon
|
7
|
+
module Schedulers
|
8
|
+
class Concurrent < Base
|
9
|
+
using RubyNext
|
10
|
+
|
11
|
+
DEFAULT_MAX_THREADS = 4
|
12
|
+
|
13
|
+
attr_reader :max_threads
|
14
|
+
|
15
|
+
def initialize(threads_count = DEFAULT_MAX_THREADS)
|
16
|
+
@max_threads = threads_count
|
17
|
+
end
|
18
|
+
|
19
|
+
def post(&block)
|
20
|
+
::Concurrent::Future.execute(executor:, &block)
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
|
25
|
+
def executor
|
26
|
+
::Concurrent::FixedThreadPool.new(max_threads)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/trellodon/schedulers.rb
CHANGED
data/lib/trellodon/version.rb
CHANGED
data/lib/trellodon.rb
CHANGED
@@ -1,8 +1,14 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "ruby-next"
|
4
|
+
|
5
|
+
require "ruby-next/language/setup"
|
6
|
+
RubyNext::Language.setup_gem_load_path(transpile: true)
|
7
|
+
|
3
8
|
require "trellodon/version"
|
4
9
|
require "trellodon/config"
|
5
|
-
require "trellodon/
|
10
|
+
require "trellodon/client"
|
11
|
+
require "trellodon/executor"
|
6
12
|
|
7
13
|
require "trellodon/entities"
|
8
14
|
require "trellodon/schedulers"
|
metadata
CHANGED
@@ -1,15 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: trellodon
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fargelus
|
8
8
|
- rinasergeeva
|
9
|
+
- palkan
|
9
10
|
autorequire:
|
10
11
|
bindir: bin
|
11
12
|
cert_chain: []
|
12
|
-
date: 2022-04-
|
13
|
+
date: 2022-04-12 00:00:00.000000000 Z
|
13
14
|
dependencies:
|
14
15
|
- !ruby/object:Gem::Dependency
|
15
16
|
name: anyway_config
|
@@ -53,12 +54,41 @@ dependencies:
|
|
53
54
|
- - ">="
|
54
55
|
- !ruby/object:Gem::Version
|
55
56
|
version: '0'
|
57
|
+
- !ruby/object:Gem::Dependency
|
58
|
+
name: concurrent-ruby
|
59
|
+
requirement: !ruby/object:Gem::Requirement
|
60
|
+
requirements:
|
61
|
+
- - ">="
|
62
|
+
- !ruby/object:Gem::Version
|
63
|
+
version: '0'
|
64
|
+
type: :runtime
|
65
|
+
prerelease: false
|
66
|
+
version_requirements: !ruby/object:Gem::Requirement
|
67
|
+
requirements:
|
68
|
+
- - ">="
|
69
|
+
- !ruby/object:Gem::Version
|
70
|
+
version: '0'
|
71
|
+
- !ruby/object:Gem::Dependency
|
72
|
+
name: ruby-next-core
|
73
|
+
requirement: !ruby/object:Gem::Requirement
|
74
|
+
requirements:
|
75
|
+
- - ">="
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: 0.15.0
|
78
|
+
type: :runtime
|
79
|
+
prerelease: false
|
80
|
+
version_requirements: !ruby/object:Gem::Requirement
|
81
|
+
requirements:
|
82
|
+
- - ">="
|
83
|
+
- !ruby/object:Gem::Version
|
84
|
+
version: 0.15.0
|
56
85
|
description: "\n The main purpose of Trellodon is to make it possible to backup
|
57
86
|
Trello boards to file system in a human-readable form (e.g., Markdown files).\n
|
58
87
|
\ "
|
59
88
|
email:
|
60
89
|
- ddraudred@gmail.com
|
61
90
|
- catherine.sergeeva@gmail.com
|
91
|
+
- dementiev.vm@gmail.com
|
62
92
|
executables:
|
63
93
|
- trellodon
|
64
94
|
extensions: []
|
@@ -70,22 +100,29 @@ files:
|
|
70
100
|
- bin/console
|
71
101
|
- bin/setup
|
72
102
|
- bin/trellodon
|
103
|
+
- lib/.rbnext/2.7/trellodon/executor.rb
|
104
|
+
- lib/.rbnext/3.0/trellodon/client.rb
|
105
|
+
- lib/.rbnext/3.1/trellodon/schedulers/concurrent.rb
|
73
106
|
- lib/trellodon.rb
|
74
|
-
- lib/trellodon/api_client.rb
|
75
|
-
- lib/trellodon/api_executor.rb
|
76
107
|
- lib/trellodon/cli.rb
|
108
|
+
- lib/trellodon/client.rb
|
77
109
|
- lib/trellodon/config.rb
|
78
110
|
- lib/trellodon/entities.rb
|
79
111
|
- lib/trellodon/entities/attachment.rb
|
80
112
|
- lib/trellodon/entities/board.rb
|
81
113
|
- lib/trellodon/entities/card.rb
|
114
|
+
- lib/trellodon/entities/checklist.rb
|
115
|
+
- lib/trellodon/entities/checklist_item.rb
|
82
116
|
- lib/trellodon/entities/comment.rb
|
83
117
|
- lib/trellodon/entities/list.rb
|
118
|
+
- lib/trellodon/entities/member.rb
|
119
|
+
- lib/trellodon/executor.rb
|
84
120
|
- lib/trellodon/formatters.rb
|
85
121
|
- lib/trellodon/formatters/base.rb
|
86
122
|
- lib/trellodon/formatters/markdown.rb
|
87
123
|
- lib/trellodon/schedulers.rb
|
88
124
|
- lib/trellodon/schedulers/base.rb
|
125
|
+
- lib/trellodon/schedulers/concurrent.rb
|
89
126
|
- lib/trellodon/schedulers/inline.rb
|
90
127
|
- lib/trellodon/schedulers/threaded.rb
|
91
128
|
- lib/trellodon/version.rb
|