@tricoteuses/senat 2.10.3 → 2.10.4

package/LICENSE.md

@@ -1,22 +1,22 @@
-# Tricoteuses-Senat
-
-## _Handle French Sénat's open data_
-
-By: Emmanuel Raviart <mailto:emmanuel@raviart.com>
-
-Copyright (C) 2019, 2020, 2021 Emmanuel Raviart
-
-https://git.tricoteuses.fr/logiciels/tricoteuses-senat
-
-> Tricoteuses-Senat is free software; you can redistribute it and/or modify
-> it under the terms of the GNU Affero General Public License as
-> published by the Free Software Foundation, either version 3 of the
-> License, or (at your option) any later version.
->
-> Tricoteuses-Senat is distributed in the hope that it will be useful,
-> but WITHOUT ANY WARRANTY; without even the implied warranty of
-> MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-> GNU Affero General Public License for more details.
->
-> You should have received a copy of the GNU Affero General Public License
-> along with this program. If not, see <http://www.gnu.org/licenses/>.
+# Tricoteuses-Senat
+
+## _Handle French Sénat's open data_
+
+By: Emmanuel Raviart <mailto:emmanuel@raviart.com>
+
+Copyright (C) 2019, 2020, 2021 Emmanuel Raviart
+
+https://git.tricoteuses.fr/logiciels/tricoteuses-senat
+
+> Tricoteuses-Senat is free software; you can redistribute it and/or modify
+> it under the terms of the GNU Affero General Public License as
+> published by the Free Software Foundation, either version 3 of the
+> License, or (at your option) any later version.
+>
+> Tricoteuses-Senat is distributed in the hope that it will be useful,
+> but WITHOUT ANY WARRANTY; without even the implied warranty of
+> MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+> GNU Affero General Public License for more details.
+>
+> You should have received a copy of the GNU Affero General Public License
+> along with this program. If not, see <http://www.gnu.org/licenses/>.

package/README.md

@@ -1,116 +1,116 @@
-# Tricoteuses-Senat
-
-## _Retrieve, clean up & handle  French Sénat's open data_
-
-## Requirements
-
-- Node >= 22
-
-## Installation
-
-```bash
-git clone https://git.tricoteuses.fr/logiciels/tricoteuses-senat
-cd tricoteuses-senat/
-```
-
-Create a `.env` file to set PostgreSQL database informations and other configuration variables (you can use `example.env` as a template). Then
-
-```bash
-npm install
-```
-
-### Database creation (not needed if downloading with Docker image)
-
-#### Using Docker
-
-```bash
-docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres
-# Default Postgres user is postgres
-# But scripts require an "opendata" role
-docker exec -it local-postgres psql -U postgres -c "CREATE ROLE opendata;"
-```
-
-## Download data
-
-Create a folder where the data will be downloaded and run the following command to download the data and convert it into JSON files.
-
-```bash
-mkdir ../senat-data/
-
-# Available options for optional `categories` parameter : All,  Ameli, Debats, DosLeg, Questions, Sens
-npm run data:download ../senat-data -- [--categories All]
-```
-
-Data from other sources is also available :
-```bash
-# Retrieval of textes and rapports from Sénat's website
-# Available options for optional `formats` parameter : xml, html, pdf
-# Available options for optional `types` parameter : textes, rapports
-npm run data:retrieve_documents ../senat-data -- --fromSession 2022 [--formats xml pdf] [--types textes]
-
-# Retrieval & parsing (textes in xml format only for now)
-npm run data:retrieve_documents ../senat-data -- --fromSession 2022 --parseDocuments
-
-# Parsing only
-npm run data:parse_textes_lois ../senat-data
-
-# Retrieval (& parsing) of agenda from Sénat's website
-npm run data:retrieve_agenda ../senat-data -- --fromSession 2022 [--parseAgenda]
-
-# Retrieval (& parsing) of comptes-rendus des débats from Sénat's website
-npm run data:retrieve_comptes_rendus ../senat-data -- [--parseDebats]
-
-# Retrieval of sénateurs' pictures from Sénat's website
-npm run data:retrieve_senateurs_photos ../senat-data
-```
-
-## Data download using Docker
-
-A Docker image that downloads and converts the data all at once is available. Build it locally or run it from the container registry.
-Use the environment variables `FROM_SESSION` and `CATEGORIES` if needed.
-
-```bash
-docker run --pull always --name tricoteuses-senat -v ../senat-data:/app/senat-data -d git.tricoteuses.fr/logiciels/tricoteuses-senat:latest
-```
-
-Use the environment variable `CATEGORIES` and `FROM_SESSION` if needed.
-
-## Using the data
-
-Once the data is downloaded, you can use loaders to retrieve it.
-To use loaders in your project, you can install the _@tricoteuses/senat_ package, and import the iterator functions that you need.
-
-```bash
-npm install @tricoteuses/senat
-```
-
-```js
-import { iterLoadSenatQuestions } from "@tricoteuses/senat/loaders"
-
-// Pass data directory and legislature as arguments
-for (const { item: question } of iterLoadSenatQuestions("../senat-data", 17)) {
-  console.log(question.id)
-}
-```
-
-## Generation of raw types from SQL schema (for contributors only)
-
-```bash
-npm run data:generate_schemas ../senat-data
-```
-
-## Publishing
-
-To publish a new version of this package onto npm, bump the package version and publish.
-
-```bash
-npm version x.y.z # Bumps version in package.json and creates a new tag x.y.z
-npx tsc
-npm publish
-```
-
-The Docker image will be automatically built during a CI Workflow if you push the tag to the remote repository.
-
-```bash
-git push --tags
-```
+# Tricoteuses-Senat
+
+## _Retrieve, clean up & handle  French Sénat's open data_
+
+## Requirements
+
+- Node >= 22
+
+## Installation
+
+```bash
+git clone https://git.tricoteuses.fr/logiciels/tricoteuses-senat
+cd tricoteuses-senat/
+```
+
+Create a `.env` file to set PostgreSQL database informations and other configuration variables (you can use `example.env` as a template). Then
+
+```bash
+npm install
+```
+
+### Database creation (not needed if downloading with Docker image)
+
+#### Using Docker
+
+```bash
+docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres
+# Default Postgres user is postgres
+# But scripts require an "opendata" role
+docker exec -it local-postgres psql -U postgres -c "CREATE ROLE opendata;"
+```
+
+## Download data
+
+Create a folder where the data will be downloaded and run the following command to download the data and convert it into JSON files.
+
+```bash
+mkdir ../senat-data/
+
+# Available options for optional `categories` parameter : All,  Ameli, Debats, DosLeg, Questions, Sens
+npm run data:download ../senat-data -- [--categories All]
+```
+
+Data from other sources is also available :
+```bash
+# Retrieval of textes and rapports from Sénat's website
+# Available options for optional `formats` parameter : xml, html, pdf
+# Available options for optional `types` parameter : textes, rapports
+npm run data:retrieve_documents ../senat-data -- --fromSession 2022 [--formats xml pdf] [--types textes]
+
+# Retrieval & parsing (textes in xml format only for now)
+npm run data:retrieve_documents ../senat-data -- --fromSession 2022 --parseDocuments
+
+# Parsing only
+npm run data:parse_textes_lois ../senat-data
+
+# Retrieval (& parsing) of agenda from Sénat's website
+npm run data:retrieve_agenda ../senat-data -- --fromSession 2022 [--parseAgenda]
+
+# Retrieval (& parsing) of comptes-rendus des débats from Sénat's website
+npm run data:retrieve_comptes_rendus ../senat-data -- [--parseDebats]
+
+# Retrieval of sénateurs' pictures from Sénat's website
+npm run data:retrieve_senateurs_photos ../senat-data
+```
+
+## Data download using Docker
+
+A Docker image that downloads and converts the data all at once is available. Build it locally or run it from the container registry.
+Use the environment variables `FROM_SESSION` and `CATEGORIES` if needed.
+
+```bash
+docker run --pull always --name tricoteuses-senat -v ../senat-data:/app/senat-data -d git.tricoteuses.fr/logiciels/tricoteuses-senat:latest
+```
+
+Use the environment variable `CATEGORIES` and `FROM_SESSION` if needed.
+
+## Using the data
+
+Once the data is downloaded, you can use loaders to retrieve it.
+To use loaders in your project, you can install the _@tricoteuses/senat_ package, and import the iterator functions that you need.
+
+```bash
+npm install @tricoteuses/senat
+```
+
+```js
+import { iterLoadSenatQuestions } from "@tricoteuses/senat/loaders"
+
+// Pass data directory and legislature as arguments
+for (const { item: question } of iterLoadSenatQuestions("../senat-data", 17)) {
+  console.log(question.id)
+}
+```
+
+## Generation of raw types from SQL schema (for contributors only)
+
+```bash
+npm run data:generate_schemas ../senat-data
+```
+
+## Publishing
+
+To publish a new version of this package onto npm, bump the package version and publish.
+
+```bash
+npm version x.y.z # Bumps version in package.json and creates a new tag x.y.z
+npx tsc
+npm publish
+```
+
+The Docker image will be automatically built during a CI Workflow if you push the tag to the remote repository.
+
+```bash
+git push --tags
+```

package/lib/databases.d.ts

@@ -19,7 +19,8 @@ type QuestionsDb = {
 type SensDb = {
     [K in keyof SensTableTypes]: SensTableTypes[K]["select"];
 };
-export declare const dbSenat: Kysely<unknown>;
+export type SenatDb = AmeliDb & DebatsDb & DoslegDb & QuestionsDb & SensDb;
+export declare const dbSenat: Kysely<SenatDb>;
 export declare const dbAmeli: Kysely<AmeliDb>;
 export declare const dbDebats: Kysely<DebatsDb>;
 export declare const dbDosleg: Kysely<DoslegDb>;

package/lib/model/ameli.d.ts

@@ -2,7 +2,7 @@ import { InferResult } from "kysely";
 export type AmendementResult = InferResult<typeof findAllQuery>[0];
 declare const findAllQuery: import("kysely").SelectQueryBuilder<{
     amd: import("../raw_types/ameli").Amd;
-    mot: import("../raw_types/ameli").Mot;
+    mot: import("kysely").Nullable<import("../raw_types/ameli").Mot>;
     sub: import("kysely").Nullable<import("../raw_types/ameli").Sub>;
     ent: import("../raw_types/ameli").Ent;
     sor: import("kysely").Nullable<import("../raw_types/ameli").Sor>;
@@ -10,13 +10,13 @@ declare const findAllQuery: import("kysely").SelectQueryBuilder<{
     avigvt: import("kysely").Nullable<import("../raw_types/ameli").Avigvt>;
     irr: import("kysely").Nullable<import("../raw_types/ameli").Irr>;
     txt_ameli: import("kysely").Nullable<import("../raw_types/ameli").TxtAmeli>;
-    typrect: import("../raw_types/ameli").Typrect;
+    typrect: import("kysely").Nullable<import("../raw_types/ameli").Typrect>;
     amdsen: import("../raw_types/ameli").Amdsen;
     sen_ameli: import("../raw_types/ameli").SenAmeli;
     grppol_ameli: import("kysely").Nullable<import("../raw_types/ameli").GrppolAmeli>;
     cab: import("kysely").Nullable<import("../raw_types/ameli").Cab>;
     com_ameli: import("kysely").Nullable<import("../raw_types/ameli").ComAmeli>;
-    etatxt: import("../raw_types/ameli").Etatxt;
+    etatxt: import("kysely").Nullable<import("../raw_types/ameli").Etatxt>;
     fbu: import("../raw_types/ameli").Fbu;
     ses: import("kysely").Nullable<import("../raw_types/ameli").Ses>;
     gvt: import("../raw_types/ameli").Gvt;
@@ -27,28 +27,41 @@ declare const findAllQuery: import("kysely").SelectQueryBuilder<{
     nat: import("kysely").Nullable<import("../raw_types/ameli").Nat>;
     sai: import("../raw_types/ameli").Sai;
     saisen: import("../raw_types/ameli").Saisen;
-    typses: import("../raw_types/ameli").Typses;
+    typses: import("kysely").Nullable<import("../raw_types/ameli").Typses>;
     typsub: import("kysely").Nullable<import("../raw_types/ameli").Typsub>;
     w_nivrec: import("../raw_types/ameli").WNivrec;
-}, "amd" | "sub" | "sor" | "avicom" | "avigvt" | "irr" | "txt_ameli" | "grppol_ameli" | "cab" | "com_ameli" | "ses" | "lec_ameli" | "nat" | "typsub", {
+}, "amd" | "mot" | "sub" | "sor" | "avicom" | "avigvt" | "irr" | "txt_ameli" | "typrect" | "grppol_ameli" | "cab" | "com_ameli" | "etatxt" | "ses" | "lec_ameli" | "nat" | "typses" | "typsub", {
     session: number | null;
+    session_libelle: string | null;
+    type_session: string | null;
     signet_dossier_legislatif: string | null;
     nature_texte: string | null;
+    nature_texte_libelle: string | null;
     numero_texte: number | null;
     numero_adoption_texte: number | null;
     intitule_texte: string | null;
+    etat_texte: string | null;
+    etat_texte_libelle: string | null;
+    type_texte: string | null;
     lecture: string | null;
     nature: string;
     id: number;
     parent_id: number | null;
     rendu_similaire_id: number | null;
     identique_id: number | null;
+    discussion_commune_id: number | null;
     numero: string | null;
     numero_absolu: number | null;
     ordre: number | null;
     accepte_gouvernement: string | null;
     subdivision_libelle: string | null;
     subdivision_libelle_court: string | null;
+    subdivision_position_texte: number | null;
+    subdivision_position_discussion: number | null;
+    subdivision_mere_id: number | null;
+    subdivision_signet: string | null;
+    subdivision_commission_id: number | null;
+    subdivision_dupliquee: string | null;
     subdivision_type: string | null;
     alinea: number | null;
     commentaire_probleme: string | null;
@@ -56,8 +69,9 @@ declare const findAllQuery: import("kysely").SelectQueryBuilder<{
     observations_additionnelles: string | null;
     date_depot: string;
     dispositif: string | null;
-    discussion_commune_id: number | null;
     objet: string | null;
+    type_rectification: string | null;
+    motion_libelle: string | null;
     etat: string;
     avis_commission: string | null;
     avis_gouvernement: string | null;
@@ -74,27 +88,44 @@ declare const findAllQuery: import("kysely").SelectQueryBuilder<{
         qualite: string | null;
         rang: number | null;
         matricule: string | null;
+        groupe_politique_id: number | null;
+        group_politique_code: string | null;
+        groupe_politique_libelle_court: string | null;
+        groupe_politique_libelle: string | null;
     }[];
 }>;
 export declare function findAll(): AsyncIterableIterator<{
     session: number | null;
+    session_libelle: string | null;
+    type_session: string | null;
     signet_dossier_legislatif: string | null;
     nature_texte: string | null;
+    nature_texte_libelle: string | null;
     numero_texte: number | null;
     numero_adoption_texte: number | null;
     intitule_texte: string | null;
+    etat_texte: string | null;
+    etat_texte_libelle: string | null;
+    type_texte: string | null;
     lecture: string | null;
     nature: string;
     id: number;
     parent_id: number | null;
     rendu_similaire_id: number | null;
     identique_id: number | null;
+    discussion_commune_id: number | null;
     numero: string | null;
     numero_absolu: number | null;
     ordre: number | null;
     accepte_gouvernement: string | null;
     subdivision_libelle: string | null;
     subdivision_libelle_court: string | null;
+    subdivision_position_texte: number | null;
+    subdivision_position_discussion: number | null;
+    subdivision_mere_id: number | null;
+    subdivision_signet: string | null;
+    subdivision_commission_id: number | null;
+    subdivision_dupliquee: string | null;
     subdivision_type: string | null;
     alinea: number | null;
     commentaire_probleme: string | null;
@@ -102,8 +133,9 @@ export declare function findAll(): AsyncIterableIterator<{
     observations_additionnelles: string | null;
     date_depot: string;
     dispositif: string | null;
-    discussion_commune_id: number | null;
     objet: string | null;
+    type_rectification: string | null;
+    motion_libelle: string | null;
     etat: string;
     avis_commission: string | null;
     avis_gouvernement: string | null;
@@ -120,6 +152,10 @@ export declare function findAll(): AsyncIterableIterator<{
         qualite: string | null;
         rang: number | null;
         matricule: string | null;
+        groupe_politique_id: number | null;
+        group_politique_code: string | null;
+        groupe_politique_libelle_court: string | null;
+        groupe_politique_libelle: string | null;
     }[];
 }>;
 export {};

package/lib/model/ameli.js

@@ -5,6 +5,7 @@ function auteurs(amendementId) {
     return jsonArrayFrom(dbAmeli
         .selectFrom("amdsen")
         .leftJoin("sen_ameli", "amdsen.senid", "sen_ameli.entid")
+        .leftJoin("grppol_ameli", "amdsen.grpid", "grppol_ameli.entid")
         .where("amdsen.amdid", "=", amendementId)
         .select([
         "amdsen.prenomuse as prenom",
@@ -13,6 +14,10 @@ function auteurs(amendementId) {
         "amdsen.qua as qualite",
         "amdsen.rng as rang",
         "sen_ameli.mat as matricule",
+        "amdsen.grpid as groupe_politique_id",
+        "grppol_ameli.cod as group_politique_code",
+        "grppol_ameli.libcou as groupe_politique_libelle_court",
+        "grppol_ameli.lilcou as groupe_politique_libelle",
     ])
         .orderBy("amdsen.rng asc"));
 }
@@ -20,10 +25,14 @@ const findAllQuery = dbAmeli
     .selectFrom("amd")
     .leftJoin("sub", "amd.subid", "sub.id")
     .leftJoin("typsub", "sub.typid", "typsub.id")
+    .leftJoin("typrect", "amd.typrectid", "typrect.id")
     .leftJoin("txt_ameli", "amd.txtid", "txt_ameli.id")
+    .leftJoin("etatxt", "txt_ameli.txtetaid", "etatxt.id")
     .leftJoin("ses", "txt_ameli.sesdepid", "ses.id")
+    .leftJoin("typses", "typses.id", "ses.typid")
     .leftJoin("nat", "txt_ameli.natid", "nat.id")
     .leftJoin("lec_ameli", "txt_ameli.lecid", "lec_ameli.id")
+    .leftJoin("mot", "amd.motid", "mot.id")
     .leftJoin("avicom", "amd.avcid", "avicom.id")
     .leftJoin("avigvt", "amd.avgid", "avigvt.id")
     .leftJoin("sor", "amd.sorid", "sor.id")
@@ -33,11 +42,17 @@ const findAllQuery = dbAmeli
     .leftJoin("cab", "amd.nomentid", "cab.entid")
     .select(({ eb, ref, val }) => [
     "ses.ann as session",
+    "ses.lil as session_libelle",
+    "typses.lib as type_session",
     "txt_ameli.doslegsignet as signet_dossier_legislatif",
     "nat.libcourt as nature_texte",
+    "nat.lib as nature_texte_libelle",
     "txt_ameli.numabs as numero_texte",
     "txt_ameli.numado as numero_adoption_texte",
     "txt_ameli.int as intitule_texte",
+    "etatxt.lic as etat_texte",
+    "etatxt.lib as etat_texte_libelle",
+    "etatxt.txttyp as type_texte",
     "lec_ameli.lib as lecture",
     eb
         .case()
@@ -54,12 +69,19 @@ const findAllQuery = dbAmeli
     "amd.amdperid as parent_id",
     "amd.amdrendusim as rendu_similaire_id",
     "amd.ideid as identique_id",
+    "amd.discomid as discussion_commune_id",
     "amd.num as numero",
     "amd.numabs as numero_absolu",
     "amd.ord as ordre",
     "amd.accgou as accepte_gouvernement",
     "sub.lib as subdivision_libelle",
     "sub.lic as subdivision_libelle_court",
+    "sub.pos as subdivision_position_texte",
+    "sub.posder as subdivision_position_discussion",
+    "sub.merid as subdivision_mere_id",
+    "sub.sig as subdivision_signet",
+    "sub.comdelid as subdivision_commission_id",
+    "sub.dupl as subdivision_dupliquee",
     "typsub.lib as subdivision_type",
     "amd.alinea as alinea",
     "amd.commentprobleme as commentaire_probleme",
@@ -67,8 +89,9 @@ const findAllQuery = dbAmeli
     "amd.mot as observations_additionnelles",
     toDateString(ref("amd.datdep")).as("date_depot"),
     "amd.dis as dispositif",
-    "amd.discomid as discussion_commune_id",
     "amd.obj as objet",
+    "typrect.lib as type_rectification",
+    "mot.lib as motion_libelle",
     eb
         .case()
         .when("amd.etaid", "=", 7)

package/lib/model/debats.d.ts

@@ -1,15 +1,6 @@
 import { InferResult } from "kysely";
 export type DebatResult = InferResult<typeof findAllQuery>[0];
-declare const findAllQuery: import("kysely").SelectQueryBuilder<{
-    debats: import("../raw_types/debats").Debats;
-    intdivers: import("../raw_types/debats").Intdivers;
-    intpjl: import("../raw_types/debats").Intpjl;
-    lecassdeb: import("../raw_types/debats").Lecassdeb;
-    secdis: import("../raw_types/debats").Secdis;
-    secdivers: import("../raw_types/debats").Secdivers;
-    syndeb: import("../raw_types/debats").Syndeb;
-    typsec: import("../raw_types/debats").Typsec;
-}, "debats", {
+declare const findAllQuery: import("kysely").SelectQueryBuilder<import("../databases").SenatDb, "debats", {
     id: string;
     date_seance: string;
     numero: number | null;

package/lib/model/debats.js

@@ -1,9 +1,10 @@
 import { jsonArrayFrom } from "kysely/helpers/postgres";
-import { dbDebats } from "../databases";
+import { dbSenat } from "../databases";
 import { ID_DATE_FORMAT } from "../scripts/datautil";
 import { toDateString } from "./util";
 function sectionsLegislatives(dateSeance) {
-    return jsonArrayFrom(dbDebats
+    return jsonArrayFrom(dbSenat
+        .withSchema("debats")
         .selectFrom("secdis")
         .leftJoin("typsec", "secdis.typseccod", "typsec.typseccod")
         .where("secdis.datsea", "=", dateSeance)
@@ -20,7 +21,8 @@ function sectionsLegislatives(dateSeance) {
         .orderBy("secdis.secdisordid asc"));
 }
 function interventionsLegislatives(sectionId) {
-    return jsonArrayFrom(dbDebats
+    return jsonArrayFrom(dbSenat
+        .withSchema("debats")
         .selectFrom("intpjl")
         .where("intpjl.secdiscle", "=", sectionId)
         .select(({ ref, val }) => [
@@ -33,7 +35,8 @@ function interventionsLegislatives(sectionId) {
         .orderBy("intpjl.intordid asc"));
 }
 function sectionsNonLegislatives(dateSeance) {
-    return jsonArrayFrom(dbDebats
+    return jsonArrayFrom(dbSenat
+        .withSchema("debats")
         .selectFrom("secdivers")
         .leftJoin("typsec", "secdivers.typseccod", "typsec.typseccod")
         .where("secdivers.datsea", "=", dateSeance)
@@ -46,7 +49,8 @@ function sectionsNonLegislatives(dateSeance) {
     ]));
 }
 function interventionsNonLegislatives(sectionId) {
-    return jsonArrayFrom(dbDebats
+    return jsonArrayFrom(dbSenat
+        .withSchema("debats")
         .selectFrom("intdivers")
         .where("intdivers.intdiverscle", "=", sectionId)
         .select(({ ref, val }) => [
@@ -59,12 +63,14 @@ function interventionsNonLegislatives(sectionId) {
         .orderBy("intdivers.intdiversordid asc"));
 }
 function lecturesAssemblee(dateSeance) {
-    return jsonArrayFrom(dbDebats
+    return jsonArrayFrom(dbSenat
+        .withSchema("debats")
         .selectFrom("lecassdeb")
         .where("lecassdeb.datsea", "=", dateSeance)
         .select("lecassdeb.lecassidt as id"));
 }
-const findAllQuery = dbDebats
+const findAllQuery = dbSenat
+    .withSchema("debats")
     .selectFrom("debats")
     .select(({ ref, val }) => [
     toDateString(ref("debats.datsea"), val(ID_DATE_FORMAT)).as("id"),