deskbot 0.1.0

data/Cargo.toml

@@ -0,0 +1,7 @@
+# This Cargo.toml is here to let externals tools (IDEs, etc.) know that this is
+# a Rust project. Your extensions dependencies should be added to the Cargo.toml
+# in the ext/ directory.
+
+[workspace]
+members = ["./ext/deskbot"]
+resolver = "2"

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Nolan J Tait
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,236 @@
+# Deskbot
+
+This is a dekstop automation library written using
+[autopilot-rs](https://github.com/autopilot-rs) and
+[rust extensions](https://bundler.io/blog/2023/01/31/rust-gem-skeleton.html)
+for Ruby.
+
+This library uses [dry-types](https://dry-rb.org/gems/dry-types/1.7/) so the
+arguments should be well documented in
+the `lib/deskbot/screen.rb` and `lib/deskbot/bitmap.rb` files
+
+The default implementation of finding images with autopilot was unbearably slow,
+so I've rewritten matching templates to use
+[opencv](https://github.com/twistedfall/opencv-rust). So ensure you have opencv
+[installed](https://github.com/twistedfall/opencv-rust/blob/master/INSTALL.md).
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+    $ bundle add deskbot
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+    $ gem install deskbot
+
+## Usage
+
+To start you can require the gem in your script and initialize a screen:
+
+```ruby
+require "deskbot"
+
+screen = Deskbot.screen
+```
+
+### Typing
+
+Type something on the keyboard
+
+```ruby
+# Type SOMETHING at 60 words per minute
+screen.type("something", flags: [:shift], wpm: 60.0, noise: 0.0)
+```
+
+You can also tap a key:
+
+```ruby
+# Tap shift + a after a 1 second delay
+screen.tap_key("a", flags: [:shift], delay_ms: 1.0)
+```
+
+And even more primitively you can toggle a key:
+
+```ruby
+# Press the "a" key down
+screen.toggle_key("a", down: true)
+```
+
+### Alerts
+
+You can make alerts
+
+```ruby
+screen.alert("Hello")
+```
+
+### Mouse movement
+
+You can teleport your mouse somewhere on the screen:
+
+```ruby
+# Teleport the mouse to coordinates x: 100, y: 100
+screen.move_mouse(100, 100)
+```
+
+You can also move the mouse smoothly somewhere:
+
+```ruby
+# Move the mouse smoothly to x: 100, y: 100 over 2 seconds
+screen.smooth_move_mouse(100, 100, duration: 2)
+```
+
+You can click:
+
+```ruby
+# Left click
+screen.click
+
+# Right click
+screen.click(:right)
+```
+
+Or even scroll:
+
+```ruby
+# Scroll 1 click up
+screen.scroll
+
+# Scroll 5 clicks up
+screen.scroll(clicks: 5)
+
+# Scroll 5 clicks down
+screen.scroll(:down, clicks: 5)
+```
+
+### Screen introspection
+
+You can query the color of a specific pixel:
+
+```ruby
+# Get the color of a specific pixel at x: 100, y: 100
+color = screen.color_at(100, 100)
+```
+
+This returns a `Deskbot::Color` object with `red`, `green`, `blue` and `alpha`
+attributes.
+
+You can query the size of the screen:
+
+```ruby
+size = screen.size
+scale = screen.scale
+```
+
+The size would be a `Deskbot::Size` with `width` and `height` attributes. The
+scale would simply be a float.
+
+You can query if a point is visible on the screen:
+
+```ruby
+screen.point_visible?(100, 100)
+screen.area_visible?(x: 100, y: 100, width: 400, height: 400)
+```
+
+### Bitmaps
+
+You can capture your screen:
+
+```ruby
+# We can capture the whole screen
+bitmap = screen.capture
+
+# Or we can capture part of our screen
+bitmap = screen.capture_area(x: 100, y: 100, width: 400, height: 400)
+```
+
+This returns a `Deskbot::Bitmap` which you can use to find areas that match
+images you provide.
+
+You can query the bounds of the bitmap:
+
+```ruby
+bitmap.bounds
+```
+
+Which would return a `Deskbot::Area` with `x`, `y`, `width` and `height`
+attributes.
+
+We can check for the colors of points on the bitmap:
+
+```ruby
+bitmap.color_at(100, 100)
+```
+
+Which returns a `Deskbot::Color`.
+
+### Comparing images
+
+You can compare images to your bitmap with a given tolerance (optional):
+
+```ruby
+bitmap.find("images/test.jpg")
+bitmap.all("images/test.jpg", tolerance: 4.0)
+```
+
+`find` and `find_color` both return `Dry::Monad::Result` objects meaning you
+need to unwrap their values:
+
+```ruby
+result = bitmap.find("images/test.jpg")
+point = result.success
+```
+
+They are also wrapped with an optional matcher so you can use a much nicer
+syntax to grab their values:
+
+```ruby
+bitmap.find("images/test.jpg") do |on|
+  on.success do |point|
+    # Do something with the point, no need to call point.success
+  end
+
+  on.failure do
+    # Handle you error here
+  end
+end
+```
+
+These return `Deskbot::Point` objects with `x` and `y` attributes.
+
+You can ask higher level questions about the image such as:
+
+```ruby
+bitmap.eql?("images/test.jpg", tolerance: 2.0)
+bitmap.count("images/test.jpg")
+```
+
+You can also query for positions of colors:
+
+```ruby
+bitmap.find_color([255, 255, 255, 0], tolerance: 0.0)
+bitmap.all_color([255, 255, 255, 0])
+```
+
+These will return `Deskbot::Point` objects with `x` and `y` attributes.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies.
+Then, run `rake spec` to run the tests. You can also run `bin/console` for
+an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+To release a new version, update the version number in `version.rb`, and then
+run `bundle exec rake release`, which will create a git tag for the version,
+push git commits and the created tag, and push the `.gem` file
+to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/nolantait/deskbot.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rb_sys/extensiontask"
+
+task build: :compile
+
+GEMSPEC = Gem::Specification.load("deskbot.gemspec")
+
+RbSys::ExtensionTask.new("deskbot", GEMSPEC) do |ext|
+  ext.lib_dir = "lib/deskbot"
+end
+
+task default: %i[compile spec]

data/ext/deskbot/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+name = "deskbot"
+version = "0.1.0"
+edition = "2021"
+authors = ["Nolan J Tait <nolanjtait@gmail.com>"]
+license = "MIT"
+publish = false
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+opencv = { version = "0.91.3", features = ["clang-runtime"] }
+magnus = { version = "0.6.2" }
+autopilot = { version = "0.4.0" }
+image = { version = "0.22.4" }

data/ext/deskbot/extconf.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require "mkmf"
+require "rb_sys/mkmf"
+
+create_rust_makefile("deskbot/deskbot")

data/ext/deskbot/src/alert.rs

@@ -0,0 +1,5 @@
+extern crate autopilot;
+
+pub fn alert(message: String) -> () {
+    autopilot::alert::alert(&message.as_str(), None, None, None);
+}

data/ext/deskbot/src/bitmap.rs

@@ -0,0 +1,268 @@
+extern crate autopilot;
+
+use image::open;
+use magnus::exception;
+use std::collections::HashMap;
+
+extern crate opencv;
+
+use opencv::{
+    core::{self, Mat, MatTraitConst, Rect, Scalar},
+    imgproc, Result,
+};
+
+#[magnus::wrap(class = "Deskbot::Providers::Autopilot::Bitmap")]
+pub struct Bitmap(autopilot::bitmap::Bitmap);
+
+impl Bitmap {
+    fn new(bitmap: autopilot::bitmap::Bitmap) -> Self {
+        Bitmap(bitmap)
+    }
+
+    pub fn save(&self, path: String) -> Result<bool, magnus::Error> {
+        match self.0.image.save(path) {
+            Ok(_) => Ok(true),
+            Err(_) => Err(magnus::Error::new(
+                exception::runtime_error(),
+                "Could not save the image",
+            )),
+        }
+    }
+
+    pub fn bounds(&self) -> HashMap<String, f64> {
+        let bounds = self.0.bounds();
+
+        HashMap::from([
+            ("x".to_string(), bounds.origin.x),
+            ("y".to_string(), bounds.origin.y),
+            ("width".to_string(), bounds.size.width),
+            ("height".to_string(), bounds.size.height),
+        ])
+    }
+
+    pub fn get_pixel(&self, x: f64, y: f64) -> Vec<u8> {
+        let point = autopilot::geometry::Point::new(x, y);
+        let value = self.0.get_pixel(point).0;
+        // For some reason I have to convert to a Vec<u8>. Magnus doesn't like [u8; 4]
+        return value.to_vec();
+    }
+
+    pub fn find_color(
+        &self,
+        color: [u8; 4],
+        tolerance: Option<f64>,
+    ) -> Option<HashMap<String, f64>> {
+        if let Some(found) = self.0.find_color(image::Rgba(color), tolerance, None, None) {
+            return Some(HashMap::from([
+                ("x".to_string(), found.x),
+                ("y".to_string(), found.y),
+            ]));
+        }
+        None
+    }
+
+    pub fn find(&self, image_path: String, tolerance: Option<f64>) -> Option<HashMap<String, f64>> {
+        let src = match opencv::imgcodecs::imread(
+            "./tmp/last-screenshot.png",
+            opencv::imgcodecs::IMREAD_COLOR,
+        ) {
+            Ok(src) => src,
+            Err(error) => panic!("Could not read the image: {}", error),
+        };
+        let templ = match opencv::imgcodecs::imread(&image_path, opencv::imgcodecs::IMREAD_COLOR) {
+            Ok(templ) => templ,
+            Err(error) => panic!("Could not read the image: {}", error),
+        };
+
+        // Create Mat for the result
+        let mut dst = Mat::default();
+        let mask = Mat::default();
+
+        // Perform template matching
+        match imgproc::match_template(
+            &src,
+            &templ,
+            &mut dst,
+            imgproc::TemplateMatchModes::TM_SQDIFF_NORMED.into(),
+            &mask,
+        ) {
+            Ok(_) => {}
+            Err(error) => panic!("Could not match the template: {}", error),
+        }
+
+        // Find the location of the best match
+        let (mut min_val, mut max_val) = (0.0, 0.0);
+        let mut min_point = core::Point { x: 0, y: 0 };
+        let mut max_point = core::Point { x: 0, y: 0 };
+
+        match core::min_max_loc(
+            &dst,
+            Some(&mut min_val),
+            Some(&mut max_val),
+            Some(&mut min_point),
+            Some(&mut max_point),
+            &mask,
+        ) {
+            Ok(_) => {}
+            Err(error) => panic!("Could not find the min max loc: {}", error),
+        }
+
+        if min_val > tolerance.unwrap_or(0.0) {
+            return None;
+        }
+
+        Some(HashMap::from([
+            ("x".to_string(), min_point.x as f64),
+            ("y".to_string(), min_point.y as f64),
+        ]))
+    }
+
+    pub fn all_color(&self, color: [u8; 4], tolerance: Option<f64>) -> Vec<HashMap<String, f64>> {
+        let mut results = vec![];
+        for found in self
+            .0
+            .find_every_color(image::Rgba(color), tolerance, None, None)
+        {
+            results.push(HashMap::from([
+                ("x".to_string(), found.x),
+                ("y".to_string(), found.y),
+            ]));
+        }
+        results
+    }
+
+    pub fn all(&self, image_path: String, tolerance: Option<f64>) -> Vec<HashMap<String, f64>> {
+        let mut results = vec![];
+
+        let mut image = self.load_image("./tmp/last-screenshot.png");
+        let template_image = self.load_image(&image_path);
+        let mut matches: Vec<core::Point> = Vec::new();
+
+        // Fake out min_val for first run through loop
+        loop {
+            match self.match_template_and_replace(
+                &mut image,
+                &template_image,
+                tolerance.unwrap_or(0.5)
+            ) {
+                Some(point) => {
+                    matches.push(point);
+                }
+                None => break,
+            }
+        }
+
+        matches.iter().for_each(|point| {
+            results.push(HashMap::from([
+                ("x".to_string(), point.x as f64),
+                ("y".to_string(), point.y as f64),
+            ]));
+        });
+
+        results
+    }
+
+    pub fn count(&self, image_path: String, tolerance: Option<f64>) -> u64 {
+        if let Ok(image) = open(image_path) {
+            let needle = autopilot::bitmap::Bitmap::new(image, None);
+            return self.0.count_of_bitmap(&needle, tolerance, None, None);
+        }
+        0
+    }
+
+    fn minimum_point(&self, mat: &Mat) -> (core::Point, f64) {
+        let mut min_val = 1.;
+        let mut min_point = core::Point { x: 0, y: 0 };
+
+        match core::min_max_loc(
+            &mat,
+            Some(&mut min_val),
+            None,
+            Some(&mut min_point),
+            None,
+            &core::no_array(),
+        ) {
+            Ok(_) => {}
+            Err(error) => panic!("Could not find the min max loc: {}", error),
+        }
+
+        (min_point, min_val)
+    }
+
+    fn load_image(&self, path: &str) -> Mat {
+        match opencv::imgcodecs::imread(path, opencv::imgcodecs::IMREAD_COLOR) {
+            Ok(src) => src,
+            Err(error) => panic!("Could not read the image: {}", error),
+        }
+    }
+
+    // fn write_image(&self, path: &str, image: &Mat) {
+    //     match opencv::imgcodecs::imwrite(path, image, &core::Vector::<i32>::new()) {
+    //         Ok(_) => {}
+    //         Err(error) => panic!("Could not write the image: {}", error),
+    //     }
+    // }
+
+    fn match_template(&self, image: &Mat, template: &Mat, threshold: f64) -> Option<core::Point> {
+        let mut result = Mat::default();
+
+        imgproc::match_template_def(image, &template, &mut result, imgproc::TM_SQDIFF_NORMED)
+            .unwrap();
+
+        let (min_point, min_val) = self.minimum_point(&result);
+
+        if min_val > threshold {
+            return None;
+        }
+
+        Some(min_point)
+    }
+
+    fn match_template_and_replace(
+        &self,
+        image: &mut Mat,
+        template: &Mat,
+        threshold: f64,
+    ) -> Option<core::Point> {
+        let min_point = self.match_template(image, template, threshold)?;
+        let template_size = template.size().unwrap();
+
+        let rect = Rect {
+            x: min_point.x,
+            y: min_point.y,
+            width: template_size.width as i32 + 1,
+            height: template_size.height as i32 + 1,
+        };
+
+        imgproc::rectangle(image, rect, Scalar::new(0.0, 0.0, 0.0, 0.0), -1, 8, 0).unwrap();
+
+        Some(min_point)
+    }
+}
+
+pub fn capture_screen_portion(x: f64, y: f64, width: f64, height: f64) -> Option<Bitmap> {
+    let point = autopilot::geometry::Point::new(x, y);
+    let size = autopilot::geometry::Size::new(width, height);
+    let rect = autopilot::geometry::Rect::new(point, size);
+    let image = autopilot::bitmap::capture_screen_portion(rect);
+
+    match image {
+        Ok(image) => {
+            image.image.save("./tmp/last-screenshot.png").unwrap();
+            Some(Bitmap::new(image))
+        }
+        Err(_) => None,
+    }
+}
+
+pub fn capture_screen() -> Option<Bitmap> {
+    let image = autopilot::bitmap::capture_screen();
+
+    match image {
+        Ok(image) => {
+            image.image.save("./tmp/last-screenshot.png").unwrap();
+            Some(Bitmap::new(image))
+        }
+        Err(_) => None,
+    }
+}

data/ext/deskbot/src/keys.rs

@@ -0,0 +1,38 @@
+use magnus::RArray;
+extern crate autopilot;
+
+fn key_flags(symbols: Vec<String>) -> Vec<autopilot::key::Flag> {
+    symbols
+        .iter()
+        .filter_map(|symbol| key_flag(symbol))
+        .collect::<Vec<autopilot::key::Flag>>()
+}
+
+fn key_flag(symbol: &String) -> Option<autopilot::key::Flag> {
+    match symbol.as_str() {
+        "shift" => Some(autopilot::key::Flag::Shift),
+        "control" => Some(autopilot::key::Flag::Control),
+        "alt" => Some(autopilot::key::Flag::Alt),
+        "meta" => Some(autopilot::key::Flag::Meta),
+        "help" => Some(autopilot::key::Flag::Help),
+        _ => None,
+    }
+}
+
+pub fn type_string(string: String, _flag: RArray, wpm: f64, noise: f64) -> () {
+    let _flags = _flag.to_vec::<String>().unwrap();
+    let flags = key_flags(_flags);
+    autopilot::key::type_string(&string, &flags, wpm, noise);
+}
+
+pub fn toggle_key(_key: char, down: bool, _flags: RArray, modifier_delay_ms: u64) -> () {
+    let flags = key_flags(_flags.to_vec::<String>().unwrap());
+    let key = autopilot::key::Character(_key);
+    autopilot::key::toggle(&key, down, &flags, modifier_delay_ms);
+}
+
+pub fn tap_key(_key: char, _flags: RArray, delay_ms: u64, modifier_delay_ms: u64) -> () {
+    let flags = key_flags(_flags.to_vec::<String>().unwrap());
+    let key = autopilot::key::Character(_key);
+    autopilot::key::tap(&key, &flags, delay_ms, modifier_delay_ms);
+}

data/ext/deskbot/src/lib.rs

@@ -0,0 +1,49 @@
+use magnus::{class, function, method, prelude::*, Error, Ruby};
+extern crate autopilot;
+
+mod bitmap;
+mod keys;
+mod mouse;
+mod screen;
+mod alert;
+
+#[magnus::init]
+fn init(ruby: &Ruby) -> Result<(), Error> {
+    let main_module = ruby.define_module("Deskbot")?.define_module("Providers")?;
+    let module = main_module.define_module("Autopilot")?;
+    module.define_singleton_method("type", function!(keys::type_string, 4))?;
+    module.define_singleton_method("toggle_key", function!(keys::toggle_key, 4))?;
+    module.define_singleton_method("tap_key", function!(keys::tap_key, 4))?;
+
+    module.define_singleton_method("mouse_location", function!(mouse::location, 0))?;
+    module.define_singleton_method("move_mouse", function!(mouse::move_to, 2))?;
+    module.define_singleton_method("toggle_mouse", function!(mouse::toggle, 2))?;
+    module.define_singleton_method("click", function!(mouse::click, 2))?;
+    module.define_singleton_method("scroll", function!(mouse::scroll, 2))?;
+    module.define_singleton_method("smooth_move_mouse", function!(mouse::smooth_move, 3))?;
+
+    module.define_singleton_method("color_at", function!(screen::get_color, 2))?;
+    module.define_singleton_method("screen_size", function!(screen::size, 0))?;
+    module.define_singleton_method("screen_scale", function!(screen::scale, 0))?;
+    module.define_singleton_method("is_point_visible", function!(screen::is_point_visible, 2))?;
+    module.define_singleton_method("is_area_visible", function!(screen::is_rect_visible, 4))?;
+
+    module.define_singleton_method("alert", function!(alert::alert, 1))?;
+
+    module.define_singleton_method(
+        "capture_screen_area",
+        function!(bitmap::capture_screen_portion, 4),
+    )?;
+    module.define_singleton_method("capture_screen", function!(bitmap::capture_screen, 0))?;
+
+    let bitmap = module.define_class("Bitmap", class::object())?;
+    bitmap.define_method("bounds", method!(bitmap::Bitmap::bounds, 0))?;
+    bitmap.define_method("find", method!(bitmap::Bitmap::find, 2))?;
+    bitmap.define_method("all", method!(bitmap::Bitmap::all, 2))?;
+    bitmap.define_method("count", method!(bitmap::Bitmap::count, 2))?;
+    bitmap.define_method("find_color", method!(bitmap::Bitmap::find_color, 2))?;
+    bitmap.define_method("all_color", method!(bitmap::Bitmap::all_color, 2))?;
+    bitmap.define_method("color_at", method!(bitmap::Bitmap::get_pixel, 2))?;
+    bitmap.define_method("save", method!(bitmap::Bitmap::save, 1))?;
+    Ok(())
+}