prune_ar 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 38a4f3eb4a02e9aabc7c1a72dc4a7324ead32471c141d09c28a5273cad6420c0
+  data.tar.gz: '08188305b0aefbefb79a056faefd2f27463a107a6b7c1e01bb626820962ab3ff'
+SHA512:
+  metadata.gz: cc89a9cf90a24df6575f6a95e27c56d2c47cb50a7d1be529f866df7476837a63346e074b2e0d9205447fc8e7c1e415455885d7367b1c36f51207aea905227d44
+  data.tar.gz: d1b0797185d44305ea7c7ae02747ae458a20676f2ae75529b1bae60e7f2c33d7cbccc3d3e771c231e8e097e1cdb600eb8a95c318c94553480453d4b7e9a83200

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased] - 2018-12-11
+
+### Added
+
+- This is the initial implementation of this gem.

data/CONTRIBUTING.md

@@ -0,0 +1,89 @@
+# Contributing
+
+When contributing to this repository, please first discuss the change you wish to make via issue,
+email, or any other method with the owners of this repository before making a change.
+
+Please note we have a code of conduct, please follow it in all your interactions with the project.
+
+## Pull Request Process
+
+1. Ensure any install or build dependencies are removed before the end of the layer when doing a
+   build.
+2. Update the README.md with details of changes to the interface, this includes new environment
+   variables, exposed ports, useful file locations and container parameters.
+3. Increase the version numbers in any examples files and the README.md to the new version that this
+   Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
+4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you
+   do not have permission to do that, you may request the second reviewer to merge it for you.
+
+## Code of Conduct
+
+### Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+### Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+- The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+### Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+### Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+### Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at devops@contently.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+### Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant homepage](http://contributor-covenant.org), version 1.4,
+available at [version](http://contributor-covenant.org/version/1/4)

data/Gemfile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in prune_ar.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,101 @@
+PATH
+  remote: .
+  specs:
+    prune_ar (0.1.0)
+      activerecord
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activemodel (5.2.2)
+      activesupport (= 5.2.2)
+    activerecord (5.2.2)
+      activemodel (= 5.2.2)
+      activesupport (= 5.2.2)
+      arel (>= 9.0)
+    activesupport (5.2.2)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 0.7, < 2)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+    arel (9.0.0)
+    ast (2.4.0)
+    coderay (1.1.2)
+    concurrent-ruby (1.1.3)
+    database_cleaner (1.7.0)
+    diff-lcs (1.3)
+    docile (1.3.1)
+    i18n (1.1.1)
+      concurrent-ruby (~> 1.0)
+    jaro_winkler (1.5.1)
+    json (2.1.0)
+    method_source (0.9.2)
+    minitest (5.11.3)
+    mysql2 (0.5.2)
+    parallel (1.12.1)
+    parser (2.5.3.0)
+      ast (~> 2.4.0)
+    pg (1.1.3)
+    powerpack (0.1.2)
+    pry (0.12.2)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    rainbow (3.0.0)
+    rake (10.5.0)
+    rspec (3.8.0)
+      rspec-core (~> 3.8.0)
+      rspec-expectations (~> 3.8.0)
+      rspec-mocks (~> 3.8.0)
+    rspec-core (3.8.0)
+      rspec-support (~> 3.8.0)
+    rspec-expectations (3.8.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-mocks (3.8.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-support (3.8.0)
+    rspec_junit_formatter (0.4.1)
+      rspec-core (>= 2, < 4, != 2.12.0)
+    rubocop (0.61.1)
+      jaro_winkler (~> 1.5.1)
+      parallel (~> 1.10)
+      parser (>= 2.5, != 2.5.1.1)
+      powerpack (~> 0.1)
+      rainbow (>= 2.2.2, < 4.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.4.0)
+    rubocop-junit_formatter (0.2)
+      rubocop (~> 0.49)
+    ruby-progressbar (1.10.0)
+    simplecov (0.16.1)
+      docile (~> 1.1)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.2)
+    sqlite3 (1.3.13)
+    thread_safe (0.3.6)
+    tzinfo (1.2.5)
+      thread_safe (~> 0.1)
+    unicode-display_width (1.4.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.17)
+  database_cleaner (~> 1.7)
+  mysql2 (~> 0.5)
+  pg (~> 1.1)
+  prune_ar!
+  pry (~> 0.12)
+  rake (~> 10.0)
+  rspec (~> 3.8)
+  rspec_junit_formatter (~> 0.4)
+  rubocop (~> 0.61)
+  rubocop-junit_formatter (~> 0.2)
+  simplecov (~> 0.12)
+  sqlite3 (~> 1.3)
+
+BUNDLED WITH
+   1.17.1

data/LICENSE

@@ -0,0 +1,7 @@
+Copyright 2018 Contently
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,240 @@
+# prune_ar
+
+[![CircleCI](https://circleci.com/gh/contently/prune_ar.svg?style=shield)](https://circleci.com/gh/contently/prune_ar)
+
+prune_ar is a gem that prunes database records using passed in deletion criteria & then pruning all subsequently orphaned records. It uses ActiveRecord's `belongs_to` associations in order to find orphaned records. It's main intent is to be able to delete any sets of records you would like to but also making sure that the database is left in a consistent state after the deletion (no orphaned records & no violated foreign key constraints). A side effect of pruning the orphaned records (done for consistency) is that it can be effectively used to prune down the whole database at once by issuing a delete on a top level table. Contently uses this process to prune down its production database down to one that is suitable for use in development (devoid of customer data).
+
+The APIs provided are **destructive** to your database, so don't run this in production.
+
+## Getting Started
+### Prerequisites
+##### Production
+No non-gem dependencies.
+
+##### Development & testing
+Make sure to have [sqlite3](https://www.sqlite.org/index.html) installed so you can run the tests.
+
+### Support
+
+#### Database
+
+The gem **should** work with any database since it uses either generic SQL or abstractions through `ActiveRecord` that should all be general. Sadly, since different databases sometimes have very different behaviors and quirks, one would expect things to possibly break on some databases. These are the databases the gem is tested against & what level of testing is done against them:
+
+Database | Level of testing (confidence of success)
+--- | ---
+PostgreSQL 9.2 | Unit tests
+PostgreSQL 9.3 | Unit tests
+PostgreSQL 9.4 | Unit tests & real world workload
+PostgreSQL 9.5 | Unit tests
+PostgreSQL 9.6 | Unit tests
+PostgreSQL 10 | Unit tests & real world workload
+PostgreSQL 11 | Unit tests
+MySQL 5 | Unit tests
+MySQL 8 | Unit tests
+MariaDB 5 | Unit tests
+MariaDB 10 | Unit tests
+SQLite 3 | Unit tests
+
+So we certainly have the highest confidence for success with PostgreSQL since we've actually tested the code against a real database. Please let us know your experience (& any issues you face) with this gem when used with your database type.
+
+If you use this gem with MySQL (or others with similar behavior), you may want to turn off the sanity checking (which creates foreign key constraints). From my small knowledge of MySQL, I believe it creates an index (if one doesn't exist) when a foreign key constraint is created. The gem does not make any effort to clean up these indexes when the foreign key constraints are deleted. It is possible that MySQL will auto-delete them? (but I'm not sure).
+
+#### ActiveRecord
+
+This gem is known to work with ActiveRecord 5.x but the version has not been fixed in the gemspec to allow you to try it out with other versions of ActiveRecord (where it may well work fine or may not).
+
+### Installation & usage
+
+For bundler, add this to your `Gemfile`:
+
+```ruby
+gem "prune_ar", "~> 0.1"
+```
+
+followed by
+
+```sh
+bundle install
+```
+
+You can also just install the gem without bundler with
+
+```sh
+gem install prune_ar
+```
+
+#### Usage
+
+Basic example:
+
+```ruby
+require 'prune_ar'
+deletion_criteria = { Account => ["accounts.internal = 'f'"] }
+Rails.application.eager_load! # We do this to make sure all models are loaded
+PruneAr::prune_all_models(deletion_criteria: deletion_criteria)
+```
+
+This will delete all external `Account`s & any child records that have an upward dependency chain (unlimited number of hops) to those deleted records. If a table does **not** have an upward dependency chain to the `Account` table, it will remain untouched.
+
+One API is provided: `PruneAr::prune_all_models`. `prune_all_models` gathers all models in your application by looking at the descendants of `ActiveRecord::Base` so it is vital that you make sure all your `ActiveRecord` models are loaded before you call this. The models gathered here are the only ones that `PruneAr` will prune orphaned records from so if only a subset of your applications models are loaded (& seen by `PruneAr`) your database could be left in an inconsistent (referential integrity wise) state after pruning or be blocked from pruning by foreign keys constraints on tables untracked by `PruneAr`.
+
+Here's a brief description of what the main parameters to these APIs mean:
+
+---
+
+##### :deletion_criteria
+The core pruning criteria that you want to execute (will be executed up front)
+```ruby
+deletion_criteria: {
+  Account => ['accounts.id NOT IN (1, 2)']
+  User => ["users.internal = 'f'", "users.active = 'f'"]
+}
+```
+
+---
+
+##### :full_delete_models
+Models for which you want to purge all records
+```ruby
+full_delete_models: [Model1, Model2]
+```
+
+---
+
+##### :pre_queries_to_run
+Arbitrary SQL statements to execute before pruning
+```ruby
+pre_queries_to_run: ['UPDATE users SET invited_by_id = NULL WHERE invited_by_id IS NOT NULL']
+```
+
+---
+
+##### :conjunctive_deletion_criteria
+Pruning criteria you want executed in conjunction with each iteration of pruning of orphaned records (one case where this is useful if pruning entities which don't have a belongs_to chain to the entities we pruned but instead are associated via join tables)
+```ruby
+conjunctive_deletion_criteria: {
+    Image => ['NOT EXISTS (SELECT 1 FROM imagings WHERE imagings.image_id = images.id)']
+}
+```
+
+---
+
+##### :perform_sanity_check (defaults to true)
+Determines whether `PruneAr` sanity checks it's own pruning by setting (& subsequently removing) foreign key constraints for all belongs_to relations. This is to prove that we maintained referential integrity.
+```ruby
+perform_sanity_check: true
+```
+
+---
+
+##### :logger
+You can provide your own logger to be used for logging any messages that the API logs
+```ruby
+logger: Logger.new(STDOUT).tap { |l| l.level = Logger::INFO }
+```
+
+---
+
+Here is an example of a rake task that is similar to what Contently uses to prune their database:
+
+```ruby
+desc 'Prune tables using prune_ar'
+task prune_tables: :no_prod_env do
+  Rails.application.eager_load!
+  PruneAr::prune_all_models(
+    deletion_criteria: {
+      Account => ['id NOT IN (589, 87)'],
+      User => ["email NOT ILIKE '%@company.com'"]
+    },
+    # This pre-query makes sure that the users that we want to keep (emails like company.com) are not pruned because they were
+    # => invited by another user that doesn't have this email (and hence we've deleted their record)
+    pre_queries_to_run: ["UPDATE users SET invited_by_id = NULL WHERE invited_by_id IS NOT NULL"],
+    # Since images are referenced via a join table, they do not have a direct upward dependency chain to another entity
+    # => so we manually prune them using the query below
+    conjunctive_deletion_criteria: {
+      Image => ['NOT EXISTS (SELECT 1 FROM imagings WHERE imagings.image_id = images.id)']
+    },
+    full_delete_models: [Comment],
+    logger: Logger.new(STDOUT).tap { |l| l.level = Logger::INFO }
+  )
+end
+```
+
+## Details
+
+The motivation for writing prune_ar came about due to two main things:
+
+- wanting a system to specify easy deletion criteria for a few top level tables & have the whole database be pruned accordingly. One use of this is to take a production database, provide a few high level table deletion criteria & end up with a pruned small clean database to be used for development purposes.
+- previous approaches to accomplish the above goal had issues with orphaned records. These orphaned records are fairly harmless on their own, but it creates major issues when one is prevented from adding foreign key constraints to these tables with orphaned records.
+
+prune_ar solves both of these. Here is a high level overview of what the algorithm used in prune_ar does:
+
+1. Gather all `belongs_to` associations for the models given.
+2. Drop all foreign key constraints (if the database supports foreign keys). This is done so prune_ar can delete records without hitting foreign key violation errors.
+3. Run any queries specified in `pre_queries_to_run`.
+4. Run the base deletion provided via `deletion_criteria`.
+5. Truncate tables for `full_delete_models`.
+6. Prune orphaned records & also delete via `conjunctive_deletion_criteria` alongside.
+7. Restore original foreign key constraints (if the database supports foreign keys).
+
+prune_ar handles polymorphic belongs_to & is able to prune [HABTM](https://guides.rubyonrails.org/association_basics.html#the-has-and-belongs-to-many-association) tables in addition simple belongs_to associations.
+
+## Development
+### Installation
+```sh
+gem install bundler
+git clone https://github.com/contently/prune_ar.git
+cd ./prune_ar
+bundle install
+```
+
+### Running the tests
+Assuming you've followed the steps outlined above for the Development installation, you can run
+```sh
+bundle exec rspec
+```
+to execute all tests.
+
+#### Test structure
+Each class has it's own `class_name_spec.rb` file in the [spec](spec) directory. The database [schema](spec/support/schema.rb) & [models](spec/support/models.rb) are located in [spec/support](spec/support).
+
+### Coding style
+Mostly standard rubocop guidelines are followed with a few modications as can be seen in [.rubocop.yml](.rubocop.yml).
+
+## Deployment
+In order to deploy a new version of the gem to rubygems:
+
+- Bump the version in [version.rb](lib/prune_ar/version.rb) as appropriate according to [SemVer](http://semver.org/).
+- Commit all changes & merge it to the master branch.
+- On latest master (after a `git pull` on master):
+
+```sh
+rake build
+rake release
+```
+
+If all went well, a new version of this gem should be published on rubygems.
+
+## Built With
+
+* [bundler](https://bundler.io/) - Dependency management.
+* [activerecord](https://rubygems.org/gems/activerecord) - This gem is based around ActiveRecord models.
+* [rspec](https://rubygems.org/gems/rspec) - Testing framework.
+
+## Contributing
+
+Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
+
+## Versioning
+
+We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/contently/prune_ar/tags).
+
+## Authors
+
+* **Anirban Mukhopadhyay** - *Initial work* - [anirbanmu](https://github.com/anirbanmu)
+
+See also the list of [contributors](https://github.com/contently/prune_ar/contributors) who participated in this project.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details