@jutge.org/toolkit 4.2.0 → 4.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/assets/prompts/ask/ask.md +2 -0
- package/dist/index.js +389 -389
- package/docs/getting-started-guide.md +12 -1
- package/docs/problem-anatomy.md +455 -0
- package/package.json +1 -1
- package/toolkit/about.ts +3 -0
- package/toolkit/ask.ts +4 -5
|
@@ -247,7 +247,18 @@ Before we dive into configuration, it's important to know how to get help:
|
|
|
247
247
|
|
|
248
248
|
Add `--help` to any command to see detailed information about its options and usage.
|
|
249
249
|
|
|
250
|
-
3. **
|
|
250
|
+
3. **Get help using JutgeAI:**
|
|
251
|
+
|
|
252
|
+
You can ask questions about the toolkit in human language using the `ask` command and get answers generated by JutgeAI and formatted in arkdown:
|
|
253
|
+
|
|
254
|
+
```bash
|
|
255
|
+
jtk ask "How to create a new problem?"
|
|
256
|
+
jtk ask "Com puc crear un problema nou?" --model "google/gemini-2.5-flash-lite"
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
For this to work, you need to have set up an AI API key (see "Setting Up AI Features" below).
|
|
260
|
+
|
|
261
|
+
4. **View information about the toolkit:**
|
|
251
262
|
|
|
252
263
|
```bash
|
|
253
264
|
jtk about
|
|
@@ -0,0 +1,455 @@
|
|
|
1
|
+
# Problem anatomy (WORK IN PROGRESS)
|
|
2
|
+
|
|
3
|
+
This document describes the anatomy of a common problem in [Jutge.org](https://jutge.org/). It explains the terminology used, the structure of a problem folder, and the purpose of each file that may be present in it.
|
|
4
|
+
|
|
5
|
+
## Terminology
|
|
6
|
+
|
|
7
|
+
A **problem** is a programming exercise meant to be solved by writing a program. All the elements of a problem are stored in a problem folder. Problems are meant to be uploaded to [Jutge.org](https://jutge.org/) where users can submit their solutions to be evaluated. The New Jutge Toolkit is a command line app to create, manage and maintain problems.
|
|
8
|
+
|
|
9
|
+
The task of the problem is described in the **problem statement**. Problem statements may be available in several human languages (Jutge.org currently supports Catalan, Spanish, French, German and English). One of them is considered the **original language** of the problem, and the others are considered **translations**. Problem statements are written in LaTeX format and are processed to generate PDF, HTML, Markdown and Text files that are shown to users. Problem statements may include text, maths, figures, code snippets, test cases, etc. Jutge.org provides LaTeX macros to organize and facilitate the writing of problem statements.
|
|
10
|
+
|
|
11
|
+
A problem contains a **golden solution**, which is a program that solves the problem correctly. The golden solution is used to generate the correct outputs for the test cases of the problem. A problem may also contain several alternative solutions in different programming languages. When a user submits a solution to a problem in some programming language, the system selects the most appropriate solution to evaluate the submission against.
|
|
12
|
+
|
|
13
|
+
A problem contains several **test cases**, each consisting of an input file and a correct output file. The test cases are used to evaluate the correctness of user submissions. Some test cases are **public** and are shown to users, while others are **private** and are hidden from users. Public test cases may be marked as **sample** test cases and are included in the problem statement to illustrate the input/output format and clarify the problem. The correct output file for each input test case is generated automatically by running the golden solution on that input file. Alternative solutions are checked against the test cases to ensure they produce the same outputs as the golden solution.
|
|
14
|
+
|
|
15
|
+
Test cases are usually written by hand or generated using **test cases generators**. A test case generator is a program that produces input files for test cases, often using randomization or specific patterns to cover different scenarios. Generator programs help create a diverse set of test cases that thoroughly test the correctness and efficiency of user submissions.
|
|
16
|
+
|
|
17
|
+
By default, the correctness of user submissions is evaluated by comparing their output files with the correct output files bit per bit. However, some problems may use custom **checkers** to evaluate the correctness of submissions. A checker is a program that takes the input file, the user's output file, and the correct output file as input and produces a verdict (Accepted, Wrong Answer, Presentation Error, etc.) based on specific criteria defined for the problem.
|
|
18
|
+
|
|
19
|
+
Some problems may include **scoring** information, which specifies how many points are awarded for passing certain test cases or groups of test cases. This allows problems to have partial credit for submissions that pass some but not all test cases.
|
|
20
|
+
|
|
21
|
+
Most problems are meant to be solved by a complete program that reads from standard input and writes to standard output. However, some problems may require users to implement specific functions or methods that will be called by a provided **main program**. In such cases, the problem specifies the function signature and behavior that users must implement. In addition, some problems may provide a **code template**, which is a code skeleton that users can fill in with their own implementation. This might be helpful to guide users on how to structure their solution, to provide utility functions or to check they are able to integrate their code correctly.
|
|
22
|
+
|
|
23
|
+
A certain class of programming languages can have their input test cases be replaced by expressions that are evaluated directly rather than read. This is the case for the RunPython, RunHaskell or RunClojure programming languages.
|
|
24
|
+
|
|
25
|
+
A problem may also contain **awards**, which are images and text that users receive when they solve the problem for the first time. These awards are meant to motivate users and recognize their achievements.
|
|
26
|
+
|
|
27
|
+
## Problem structure
|
|
28
|
+
|
|
29
|
+
A problem is a folder with `.pbm` extension that can be structured in two ways:
|
|
30
|
+
|
|
31
|
+
- Method 1: test cases and solutions are the same for all human languages (**preferred**)
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
└── problem_folder.pbm
|
|
35
|
+
├── problem.yml
|
|
36
|
+
├── handler.yml
|
|
37
|
+
├── problem.ca.tex
|
|
38
|
+
├── problem.ca.yml
|
|
39
|
+
├── problem.en.tex
|
|
40
|
+
├── problem.en.yml
|
|
41
|
+
├── sample.inp
|
|
42
|
+
├── sample.cor
|
|
43
|
+
├── solution.cc
|
|
44
|
+
└── ...
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
- Method 2: different test cases or solutions for different human languages (**discouraged**)
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
└── problem_folder.pbm
|
|
51
|
+
├── ca
|
|
52
|
+
| ├── handler.yml
|
|
53
|
+
| ├── problem.ca.tex
|
|
54
|
+
| ├── problem.ca.yml
|
|
55
|
+
| ├── sample.inp
|
|
56
|
+
| ├── sample.cor
|
|
57
|
+
| ├── solution.cc
|
|
58
|
+
| └── ...
|
|
59
|
+
├── en
|
|
60
|
+
| ├── handler.yml
|
|
61
|
+
| ├── problem.en.tex
|
|
62
|
+
| ├── problem.en.yml
|
|
63
|
+
| ├── sample.inp
|
|
64
|
+
| ├── sample.cor
|
|
65
|
+
| ├── solution.cc
|
|
66
|
+
| └── ...
|
|
67
|
+
└── problem.yml
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
The above structures are just two examples of the structure that a problem can have and therefore should only be considered as a guideline. The purpose of all the files is explained later on this file.
|
|
71
|
+
|
|
72
|
+
The first method is preferred whenever possible, as it avoids duplication of files and is easier to maintain. The second method is discouraged and should be used only when absolutely necessary, for instance when the test cases or the solutions differ between languages.
|
|
73
|
+
|
|
74
|
+
## Problem files TODO
|
|
75
|
+
|
|
76
|
+
A problem should contain the following files:
|
|
77
|
+
|
|
78
|
+
- `solution.*`: One or more solutions to the problem in different languages. See [Solutions](#solutions) for more information.
|
|
79
|
+
- `handler.yml`: Contains the information of how to handle the problem. See [Handler](#handler) for more information.
|
|
80
|
+
- `tags.yml`: Contains all the tags associated to the problem as a YAML list of words. Only for instructors, can be left empty. See [Tags](#tags) for more information.
|
|
81
|
+
- `*.inp`: All the input test sets. See [Test cases](#test-cases) for more information.
|
|
82
|
+
- `*.cor`: All the correct files. Those are generated automatically by the toolkit from the `solution.*` file. See [Test cases](#test-cases) for more information.
|
|
83
|
+
- `problem.lang.tex`: Statement LaTeX file for language `lang`. See [Problem statement](#problem-statement) for more information.
|
|
84
|
+
- `problem.lang.yml`: Contains the problem information in language `lang`. See [Problem metadata](#problem-metadata) for more information.
|
|
85
|
+
- `problem.lang.pdf`: Formatted PDF statement for language `lang`. These are generated automatically by the toolkit.
|
|
86
|
+
|
|
87
|
+
Additionally, the problem can contain the following optional files:
|
|
88
|
+
|
|
89
|
+
- `award.png`: Image of the award users will obtain when they get the problem accepted for the first time. See [Awards](#awards) for more information.
|
|
90
|
+
- `award.html`: HTML description of the award users will obtain when they get the problem accepted for the first time. See [Awards](#awards) for more information.
|
|
91
|
+
- `code.*`: Code provided in the problem to users that is given as a solution template (a function with blanks to be filled, for example).
|
|
92
|
+
- `distiller.yml`: File used to specify the parameters of the distillation process. See [Distilled test cases](#distilled-test-cases) for more information.
|
|
93
|
+
- `scores.yml`: File that describes the scoring of a problem. See [Scoring](#scoring) for more information.
|
|
94
|
+
- `*.ops`: File used to specify some limits for the correction of the problem. See [Test options](#test-options) for more information.
|
|
95
|
+
|
|
96
|
+
## Problem statement
|
|
97
|
+
|
|
98
|
+
Problem statement are stored in LaTeX files named `problem.lang.tex`, where `lang` denotes the ISO 639-1 code of the language for which the metadata is given (`ca`, `en`, `es`, `fr`, `de` currently). Problem statements make use of certain macros defined by [Jutge.org](https://jutge.org/).
|
|
99
|
+
|
|
100
|
+
### Structure
|
|
101
|
+
|
|
102
|
+
The typical structure of a problem statement is the following:
|
|
103
|
+
|
|
104
|
+
```latex
|
|
105
|
+
\Problem{The title of the problem using LaTeX}
|
|
106
|
+
|
|
107
|
+
\Statement
|
|
108
|
+
|
|
109
|
+
This section provides the statement of the problem.
|
|
110
|
+
|
|
111
|
+
\medskip
|
|
112
|
+
|
|
113
|
+
Different paragraphs should be separated by the \medskip command.
|
|
114
|
+
|
|
115
|
+
\Input
|
|
116
|
+
|
|
117
|
+
This section describes the input of the problem.
|
|
118
|
+
|
|
119
|
+
\Output
|
|
120
|
+
|
|
121
|
+
This section describes the output of the problem.
|
|
122
|
+
|
|
123
|
+
\Sample
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
The `\Sample` section will be automatically replaced to contain the sample test cases. Alternatively, it is possible to adjust the formatting of the sample test cases: `\SampleOneCol` places input and output test cases one below the other, while `\SampleTwoCol` places them side by side.
|
|
128
|
+
|
|
129
|
+
`RunPython` users should use `\SampleSession` to get their sample test cases properly formatted as interactive Python sessions. [CHECK]
|
|
130
|
+
|
|
131
|
+
The title inside the `\Problem{}` macro should match the title in the metadata given in the `problem.lang.yml` file, but the title on the LaTeX file can contain math or LaTeX macros.
|
|
132
|
+
|
|
133
|
+
### Figures
|
|
134
|
+
|
|
135
|
+
Figures can be inserted using the `\FigureL`, `\FigureC` and `\FigureR` macros, which stand for _left_, _center_ and _right_ respectively. These macros have two parameters:
|
|
136
|
+
|
|
137
|
+
1. The formatting of the figure (as in `\includegraphics[...]`).
|
|
138
|
+
|
|
139
|
+
2. The filename of the figure, with no extension. The figure should exist in JPG or PNG format [CHECK]. The use of SVG format might be possible [CHECK]. The use of EPS format is deprecated.
|
|
140
|
+
|
|
141
|
+
For instance, `\FigureR{width=4.5cm}{image}` will place figure `image` to the right with a width of 4.5 centimeters.
|
|
142
|
+
|
|
143
|
+
### Quotes
|
|
144
|
+
|
|
145
|
+
In order to enclose texts between quotes, please use the `\q{...}` (single quotes), `\qq{...}` (double quotes) or `\uq{...}` (no quotes, but same style) macros. [TODO: we have to honor/improve this]
|
|
146
|
+
|
|
147
|
+
### Code
|
|
148
|
+
|
|
149
|
+
The `lstlisting` macros may be used in order to include code. Short snippets of code can be written between `@` signs [CHECK].
|
|
150
|
+
|
|
151
|
+
By default, C++ style is used. It may be changed using standard `lstlisting` definitions. The `\UseHaskell` and `\UsePython` macros are shortcuts to use Haskell or Python styling respectively.
|
|
152
|
+
|
|
153
|
+
### Scoring
|
|
154
|
+
|
|
155
|
+
For problems with partial scoring based on passing test cases, use the `\Scoring` macro followed by an explaination of the scoring method (for example, how many points each test case is worth).
|
|
156
|
+
|
|
157
|
+
### Other sectioning macros
|
|
158
|
+
|
|
159
|
+
There exist a few other macros that may be used in various situations. Their effect should be straightforward:
|
|
160
|
+
|
|
161
|
+
- `\Precondition`
|
|
162
|
+
- `\Observation`
|
|
163
|
+
- `\Observations`
|
|
164
|
+
- `\Specification`
|
|
165
|
+
- `\Interface`
|
|
166
|
+
- `\Hint`
|
|
167
|
+
- `\Scores` [CHECK]
|
|
168
|
+
- `\Tuples`
|
|
169
|
+
- `\ObservationElastic`
|
|
170
|
+
- `\ObservationElasticII`
|
|
171
|
+
- `\ObservationNoMain`
|
|
172
|
+
- `\ObservationNoMainPlural`
|
|
173
|
+
- `\ObservationNoMainTuples` [CHECK]
|
|
174
|
+
- `\ObservationNoMainTuplesPlural` [CHECK]
|
|
175
|
+
- `\ObservationNoMainClasses` [CHECK]
|
|
176
|
+
|
|
177
|
+
### Other macros
|
|
178
|
+
|
|
179
|
+
The `\CPP` macro prints C++ in a beautiful way.
|
|
180
|
+
|
|
181
|
+
The `\Link{...}` macro provides an hyperlink to another problem (without specified language), e.g., `\Link{P68688}`.
|
|
182
|
+
|
|
183
|
+
## Problem metadata
|
|
184
|
+
|
|
185
|
+
Problem metadata is stored using YAML syntax in files named `problem.lang.yml`, where `lang` denotes the ISO 639-1 code of the language for which the metadata is given (`en`, `es`, `ca`, ...).
|
|
186
|
+
|
|
187
|
+
In the case that `lang` denotes the original language of the problem, `problem.lang.yml` should contain the following fields:
|
|
188
|
+
|
|
189
|
+
- `title`: Title of the problem in the original language.
|
|
190
|
+
- `author`: Full name of the problem setter.
|
|
191
|
+
- `email`: Email of the problem setter.
|
|
192
|
+
|
|
193
|
+
If `lang` denotes a translation, `problem.lang.yml` should contain the following fields:
|
|
194
|
+
|
|
195
|
+
- `translator`: Full name of the problem translator.
|
|
196
|
+
- `title`: Title of the problem in the translated language.
|
|
197
|
+
- `translator_email`: Email of the problem translator.
|
|
198
|
+
- `original_language`: Code for the original language of the problem.
|
|
199
|
+
|
|
200
|
+
A problem should have one unique original language and may have several translations. All translations should refer to the same original language. The system will find the original author through it.
|
|
201
|
+
|
|
202
|
+
All the values for the metadata fields should be given as Unicode based plain text. Remember to enclose them in quotes if they contain special characters such as `:` or `'`.
|
|
203
|
+
|
|
204
|
+
### Examples
|
|
205
|
+
|
|
206
|
+
`problem.ca.yml`:
|
|
207
|
+
|
|
208
|
+
```yml
|
|
209
|
+
title: Suma de dos enters
|
|
210
|
+
author: Jordi Petit
|
|
211
|
+
email: jpetit@somewhere.mail.com
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
`problem.en.yml`:
|
|
215
|
+
|
|
216
|
+
```yml
|
|
217
|
+
title: Sum of two integer numbers
|
|
218
|
+
translator: Carlos Molina
|
|
219
|
+
translator_email: cmolina@somewhere.mail.com
|
|
220
|
+
original_language: ca
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
## The `problem.yml` file
|
|
224
|
+
|
|
225
|
+
Once a problem is uploaded to [Jutge.org](https://jutge.org/), a `problem.yml` file is automatically created in the root of the problem folder. This file contains metadata about the problem that is used by the system. The fields of this file should not be modified by hand. In it, you will find fields such as the problem identifier (eg, X12345), its passcode, the time of creation, and the time of the last modification. Importantly, the presence of this file indicates that the problem has already been uploaded to [Jutge.org](https://jutge.org/).
|
|
226
|
+
|
|
227
|
+
If you are creating a problem from a copy of an existing problem, be sure to delete the `problem.yml` file before uploading it to [Jutge.org](https://jutge.org/) to not interfere with the existing problem.
|
|
228
|
+
|
|
229
|
+
## Handler TODO
|
|
230
|
+
|
|
231
|
+
The file `handler.yml` contains the information of how to handle the problem using YAML syntax. These options will tell the toolkit how to compile the problem. The handler must have two options:
|
|
232
|
+
|
|
233
|
+
- `handler`: `std` (default option) and `graphic` (used by some Python3 problems).
|
|
234
|
+
- `source_modifier`: `none` (default option), `structs` (C++ problems that use structs), `no_main` (problems where only a function is requested, in that case you will need to use the optional option `func_name`).
|
|
235
|
+
|
|
236
|
+
There are also optional arguments that may be used:
|
|
237
|
+
|
|
238
|
+
- `checker`: Option that will tell [Jutge.org](https://jutge.org/) how to compare the user's program output with the problem solution. See [Checker](#checker) for more information.
|
|
239
|
+
|
|
240
|
+
- `compilers`: Compiler that [Jutge.org](https://jutge.org/) will use to correct the problem. If specified, this will be the only compiler available on the website if an user wants to send his submission. You will find a list of available compilers in https://jutge.org/documentation/compilers.
|
|
241
|
+
|
|
242
|
+
- `func_name`: Name of the function requested in case the problem only asks for a function (this must be used along with `source_modifier: no_main`)
|
|
243
|
+
|
|
244
|
+
- `invisible_main`: If set to `1`, the `main.cc` file will not be provided to users. This is used by problems that do not require to write a `main` function.
|
|
245
|
+
|
|
246
|
+
- `presentation_error`: If set to `1 ` (by default), the PE (Presentation Error) verdict will be enabled. Otherwise, the verdict will be WA (Wrong Answer) if the files are not identical.
|
|
247
|
+
|
|
248
|
+
- `solution`: Programming language of the solution that will be used to generate the correct output test cases. By default, if a compiler is specified under `compilers` it will use that language. Otherwise, it will use the C++ solution.
|
|
249
|
+
|
|
250
|
+
- `pylibs`: Problems in Python can allow the usage of certain non standard Python libraries. `pylibs` is a list of libraries that may be used to solve the current problem. For instance, `pylibs: [numpy, networkx]`. By default, this list is empty. Available libraries are in https://jutge.org/documentation/pylibs.
|
|
251
|
+
|
|
252
|
+
By default, the content of `handler.yml ` should be like this:
|
|
253
|
+
|
|
254
|
+
```
|
|
255
|
+
handler: std
|
|
256
|
+
source_modifier: none
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
### Limiting the maximum time a submission can take
|
|
260
|
+
|
|
261
|
+
There are two arguments that can be used to determine the maximum time that a submission can take depending on the time the solution took to solve the problem. The submission will have a maximum time to complete the task which is calculated with the following function:
|
|
262
|
+
|
|
263
|
+
`Maximum submission time: solution_time * time_factor + time_constant.`
|
|
264
|
+
|
|
265
|
+
`time_factor` and `time_constant` are arguments that can be specified on the `handler.yml` file. They are a list with a maximum of three values, one for each type of compilers in the following order:
|
|
266
|
+
|
|
267
|
+
- Fast: Ada, C, C++, D, Fortran, Go, Haskell, Pascal
|
|
268
|
+
- Medium: FreeBASIC, C#, Java, Guile
|
|
269
|
+
- Slow: Brainfuck, Erlang, JavaScript, CLISP, Lua, PHP, Perl, Python, R, Ruby and Whitespace
|
|
270
|
+
|
|
271
|
+
If one or two values are missing, they will be calculated using the following formulas: `medium = fast * 5`; `slow = medium * 2`. This applies for both `time_factor` and `time_constant` arguments.
|
|
272
|
+
|
|
273
|
+
If the arguments have not been specified, the default `time_factor` and `time_constant` values for all the compilers will be `2` and `0.1` respectively.
|
|
274
|
+
|
|
275
|
+
This is an example of how the arguments would look like in `handler.yml `:
|
|
276
|
+
|
|
277
|
+
```handler.yml
|
|
278
|
+
time_factor: [2, 10, 2]
|
|
279
|
+
time_constant: [0.1, 0.01, 0.05]
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
## Test cases
|
|
283
|
+
|
|
284
|
+
Each test case `test` is described through two files: `test.inp` and `test.cor`.
|
|
285
|
+
|
|
286
|
+
`test.inp` contains the input of the test case and `test.cor` contains the correct output of the case. In addition, `test` can also make use of a `test.ops` file to describe some options for its correction.
|
|
287
|
+
|
|
288
|
+
Test case names should be made of letters, digits, dashes and underscores. Do not use special characters in them. Test cases whose name starts with `sample`, `public`, `hint` or `distilled` have an special meaning and will be explained later on. In the case several of these must be present, they often get names such as `sample-1`, `sample-2`, or `sample-short`.
|
|
289
|
+
|
|
290
|
+
At correction time, public test cases are corrected before the private ones. The cases are processed sequentially, ordered by their name.
|
|
291
|
+
|
|
292
|
+
As correcting each single test case causes some overhead, it is not advisable to have many different test cases. 20 different test cases may be a reasonable limit.
|
|
293
|
+
|
|
294
|
+
Input and output test cases should follow these rules:
|
|
295
|
+
|
|
296
|
+
- They only contain common characters: digits, uppercase and lowercase letters, punctuation ... (specifically those with ASCII codes between 32 and 126, plus the newline). So, no accents such as `À` or `ü`, no characters such as `ç`, `Δ`, `市` or `😊`, and no tabs.
|
|
297
|
+
|
|
298
|
+
- All lines should end with a newline character (`\n`), including the last line.
|
|
299
|
+
|
|
300
|
+
In addition, these rules should be taken into account in the output files:
|
|
301
|
+
|
|
302
|
+
- There must be no spaces before a line break. In particular, there must be no lines with only one or more spaces before the line break.
|
|
303
|
+
|
|
304
|
+
### Sample test cases
|
|
305
|
+
|
|
306
|
+
Sample test cases start with `sample` and will be revelaed to users in the problem statement. As such, they should make clear the format of the input and the output for the problem and should be reasonably short.
|
|
307
|
+
|
|
308
|
+
#### Public test cases
|
|
309
|
+
|
|
310
|
+
Public test cases start with `public` and will be revelead to users, but will not be shown in the problem statement. Usage of public test cases should be rare, but can be useful in situations where long input/output samples must be delivered.
|
|
311
|
+
|
|
312
|
+
#### Hint test cases
|
|
313
|
+
|
|
314
|
+
Hint test cases start with `hint` and will be revealed to users if the submission is not accepted (unless some sample test case also fails).
|
|
315
|
+
|
|
316
|
+
#### Distilled test cases
|
|
317
|
+
|
|
318
|
+
Distilled test cases start with `distilled` and will be shown to users whose submission fails on them (unless some sample or hint test case also fails). Distilled test cases are created by the system using the [Distiller Algorithm](http://upcommons.upc.edu/handle/2117/28174) and are added automatically to the problem directory, they are not mean to be created or modified by hand.
|
|
319
|
+
|
|
320
|
+
File `distiller.yml` is used to specify the parameters of the distillation process. `distillation.yml` is used to get some statistics form the distillation process.
|
|
321
|
+
|
|
322
|
+
Unfortunately, the distillation process requires some hand work and is not currently fully automated.
|
|
323
|
+
|
|
324
|
+
### Test options
|
|
325
|
+
|
|
326
|
+
The `test.ops` file can be used to specify some limits for the correction of the corresponding test case. The options should be written as if they were arguments of a program, with a space between each other. The following options are available:
|
|
327
|
+
|
|
328
|
+
- `--verbose`: Sets the verbose output level. It has three options: `0` (quiet), `1` (normal) and `2` (all the information).
|
|
329
|
+
|
|
330
|
+
- `--maxtime`: Sets maximum execution time. It has three values: `cputime`, `limtime` and `clktime`. You can specify only the `cputime`, the `cputime` and the`limtime` or all of them.
|
|
331
|
+
- `cputime` is the maximum time that the program has to use the CPU (processing instructions).
|
|
332
|
+
|
|
333
|
+
- `limtime` is the maximum CPU time limit. If not specified, `limtime` will be `cputime+1.5`.
|
|
334
|
+
|
|
335
|
+
- `clktime` is the maximum clock time (total time) the program has to run. If not specified, `clktime` will be three times `limtime`.
|
|
336
|
+
|
|
337
|
+
- `--maxmem`: Set max memory in MB. It has two values: `maxmem` and `maxstack`. `maxmem` is the maximum amount of memory that the program is allowed to use. `maxstack` is the maximum amount of `maxmem` that can be used for the stack. If the `maxstack` is not specified, it will be the same as `maxmem`.
|
|
338
|
+
|
|
339
|
+
- `--maxfiles`: Sets the maximum number of files that can be open simultaneously.
|
|
340
|
+
|
|
341
|
+
- `--maxprocs`: Sets the maximum number of processes that the program is allowed to create.
|
|
342
|
+
|
|
343
|
+
- `--maxoutput`: Sets the maximum file size in MB of the output generated by the program.
|
|
344
|
+
|
|
345
|
+
- `--maxcore`: Sets the maximum file size in MB of the core file (generated if the program crashes).
|
|
346
|
+
|
|
347
|
+
- `--basename`: Sets the base name of the input test set.
|
|
348
|
+
|
|
349
|
+
- `--stdout`: Set path of the output (`.out`) file. If not specified, the output name will be `basename.out`.
|
|
350
|
+
|
|
351
|
+
- `--stdin`: Set path of the input (`.inp`) file. If not specified, the output name will be `basename.inp`.
|
|
352
|
+
|
|
353
|
+
- `--stderr`: Set path of the error (`.err`) file. If not specified, the output name will be `basename.err`.
|
|
354
|
+
|
|
355
|
+
- `--logfile`: Set path of the `.log` file. If not specified, the output name will be `basename.log`.
|
|
356
|
+
|
|
357
|
+
- `--resfile`: Set path of the `.res` file. If not specified, the output name will be `basename.res`.
|
|
358
|
+
|
|
359
|
+
## Solutions
|
|
360
|
+
|
|
361
|
+
Each problem must have, one golden solution and may have several alternative solutions in different programming languages. By default, the golden solution is C++, if you want to change it you can specify it in the `handler.yml` file with the `solution` field (eg, `solution: Python3`). Currently supported programming languages for solutions are C, C++, Java, Python3, Haskell, Clojure, Rust, RunPython, RunHaskell and RunClojure.
|
|
362
|
+
|
|
363
|
+
Independently of the available solutions, users can submit their solutions in any supported programming language. The system will match the programming language of the submission and the programming languages available for the solution and select the most appropriate one.
|
|
364
|
+
|
|
365
|
+
Solution files are named `solution.ext`, where `ext` is the standard extension that corresponds to the selected programming language. For instance, a problem may contain `solution.cc` and `solution.py` in order to provide reference solutions in C++ and Python3.
|
|
366
|
+
|
|
367
|
+
Java solutions must have their `main` function be placed in a `Main` class. See https://jutge.org/documentation/compilers/JDK for some examples.
|
|
368
|
+
|
|
369
|
+
## Checker
|
|
370
|
+
|
|
371
|
+
With the `checker` option you can specify how the user's program output is compared with the problem solution. This is especially useful if the order of the output does not matter, for example. There are various `checker` options:
|
|
372
|
+
|
|
373
|
+
- `std`: The default option, it will just compare the files normally.
|
|
374
|
+
|
|
375
|
+
- `loosy`: Used for loosy outputs. It is like the standard checker but it will return AC (Accepted) in case a problem gets a PE (Presentation Error) verdict.
|
|
376
|
+
|
|
377
|
+
- `epsilon`: Used for problems where the output are numbers and can be slightly different than the problem solution. It will check if the user solution is similar to the problem solution. This checker has the following options:
|
|
378
|
+
- `relative`: If set to `1`, the comparison method will be relative. Otherwise, it will be absolute.
|
|
379
|
+
|
|
380
|
+
- `elastic`: Used to check for outputs whose order is independent. This happens, for instance, for backtracking problems where output order is left without specification. This checker has the following options:
|
|
381
|
+
- `separator`: String that separates one case from another one.
|
|
382
|
+
|
|
383
|
+
- `elastic2`: Used to check for outputs whose order is independent and the order of these outputs are also independent. This happens, for instance, for subset backtracking problems where output order is left without specification and the order of the subset is also unordered. This checker has the following options:
|
|
384
|
+
- `separator1`: String that separates one case from another one.
|
|
385
|
+
|
|
386
|
+
- `separator2`: String that separates one case inside the case from another one.
|
|
387
|
+
|
|
388
|
+
- `starting`: String that starts one case.
|
|
389
|
+
|
|
390
|
+
- `ending`: String that ends one case.
|
|
391
|
+
|
|
392
|
+
- `external`: used to check for outputs using an external program that reads the input and the generated output and writes to `stdout` the verdict. This checker has the following options:
|
|
393
|
+
- `external_program`: Name of the external program used. If the program does not exist, an IE (Internal Error) verdict will be returned.
|
|
394
|
+
|
|
395
|
+
- `external_timeout`: Time that the external program has to do its work. The default time is 5 seconds. An IE (Internal Error) verdict will be returned it the program runs for more than the timeout.
|
|
396
|
+
|
|
397
|
+
## Scores
|
|
398
|
+
|
|
399
|
+
In order to score submissions according to the correct test cases it passes, the file `scores.yml` must exist. `scores.yml` describes the scoring of a problem using YAML syntax. The scores are given through a list of partial scores. Each partial score contains the following fields:
|
|
400
|
+
|
|
401
|
+
- `part`: Identifier of the partial score.
|
|
402
|
+
- `prefix`: Prefix of the test cases that must be passed in this partial score.
|
|
403
|
+
- `points`: Number of points assigned to this partial score.
|
|
404
|
+
|
|
405
|
+
The total number of points is usually 100, but other (integer) values can be used. A submission that receives the totality of points is considered accepted.
|
|
406
|
+
|
|
407
|
+
### Example
|
|
408
|
+
|
|
409
|
+
Consider a problem that has the following test cases:
|
|
410
|
+
|
|
411
|
+
- `sample-1`
|
|
412
|
+
- `sample-2`
|
|
413
|
+
- `easy-A`
|
|
414
|
+
- `easy-B`
|
|
415
|
+
- `hard-A`
|
|
416
|
+
- `hard-B`
|
|
417
|
+
- `hard-C`
|
|
418
|
+
|
|
419
|
+
The following scores file gives 10 points to submissions passing all sample test cases, 30 points to submissions passing all easy test cases, and 60 points to submissions passing all hard test cases:
|
|
420
|
+
|
|
421
|
+
`scores.yml`
|
|
422
|
+
|
|
423
|
+
```yml
|
|
424
|
+
- part: Samples
|
|
425
|
+
prefix: sample
|
|
426
|
+
points: 10
|
|
427
|
+
- part: Easy
|
|
428
|
+
prefix: easy
|
|
429
|
+
points: 30
|
|
430
|
+
- part: Hard
|
|
431
|
+
prefix: hard
|
|
432
|
+
points: 60
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
Overlapping prefixes can be used to create more complex scoring schemes.
|
|
436
|
+
|
|
437
|
+
## Awards
|
|
438
|
+
|
|
439
|
+
[Jutge.org](https://jutge.org/) offers awards to users in specific circumstances. Awards are images with a caption and a short description. In the case that a problem contains an image file `award.png` and (optionally) an HTML file `award.html`, users who get the problem accepted for the first time will receive the award.
|
|
440
|
+
|
|
441
|
+
The `award.png` image file should be a square image in PNG format with a transparent or white background (preferably) of size 200x200 pixels or more. Clip art and colorful images are preferred, no offensive images should be used.
|
|
442
|
+
|
|
443
|
+
The `award.html` file should contain a description of the award using simple HTML code. If `award.html` is missing but `award.png` exists, a default description will be provided by the system.
|
|
444
|
+
|
|
445
|
+
## Quick checklist
|
|
446
|
+
|
|
447
|
+
- Make sure that all file and folder names only contain letters, digits, dots, dashes and underscores. Do not use special characters, blanks or accents in them.
|
|
448
|
+
|
|
449
|
+
- Make sure that all lines in input and output test cases end with a newline character (`\n`), including the last line.
|
|
450
|
+
|
|
451
|
+
- Make sure that there are no spaces before a line break in output test cases. In particular, there must be no lines with only one or more spaces before the line break.
|
|
452
|
+
|
|
453
|
+
- Make sure that all files are encoded in UTF-8.
|
|
454
|
+
|
|
455
|
+
- Make sure that all test cases files are encoded in 7-bit ASCII.
|
package/package.json
CHANGED
package/toolkit/about.ts
CHANGED
|
@@ -21,6 +21,9 @@ export const aboutCmd = new Command('about')
|
|
|
21
21
|
for (const contributor of packageJson.contributors!) {
|
|
22
22
|
showPerson(contributor)
|
|
23
23
|
}
|
|
24
|
+
tui.print('')
|
|
25
|
+
tui.print('Documentation:')
|
|
26
|
+
tui.url('https://github.com/jutge-org/new-jutge-toolkit/tree/main/docs')
|
|
24
27
|
})
|
|
25
28
|
|
|
26
29
|
function showPerson(person: string | { name: string; email?: string; url?: string }) {
|
package/toolkit/ask.ts
CHANGED
|
@@ -14,10 +14,7 @@ export const askCmd = new Command('ask')
|
|
|
14
14
|
.option('-m, --model <model>', 'AI model to use', settings.defaultModel)
|
|
15
15
|
|
|
16
16
|
.action(async (question, { model }) => {
|
|
17
|
-
|
|
18
|
-
askCmd.help()
|
|
19
|
-
return
|
|
20
|
-
}
|
|
17
|
+
console.log()
|
|
21
18
|
|
|
22
19
|
const systemPrompt = await readTextInDir(join(projectDir(), 'assets', 'prompts', 'ask'), 'ask.md')
|
|
23
20
|
|
|
@@ -26,7 +23,9 @@ export const askCmd = new Command('ask')
|
|
|
26
23
|
|
|
27
24
|
const answer = await complete(model, systemPrompt, fullPrompt)
|
|
28
25
|
await tui.markdown(answer)
|
|
29
|
-
tui.warning(
|
|
26
|
+
tui.warning(
|
|
27
|
+
`This answer generated by JutgeAI using ${model} is not authoritative but we hope it will help you.`,
|
|
28
|
+
)
|
|
30
29
|
})
|
|
31
30
|
|
|
32
31
|
export async function loadDocumentation(): Promise<string> {
|