AXElements 0.6.0beta1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.yardopts +20 -0
- data/LICENSE.txt +25 -0
- data/README.markdown +150 -0
- data/Rakefile +109 -0
- data/docs/AccessibilityTips.markdown +119 -0
- data/docs/Acting.markdown +340 -0
- data/docs/Debugging.markdown +326 -0
- data/docs/Inspecting.markdown +255 -0
- data/docs/KeyboardEvents.markdown +57 -0
- data/docs/NewBehaviour.markdown +151 -0
- data/docs/Notifications.markdown +271 -0
- data/docs/Searching.markdown +250 -0
- data/docs/TestingExtensions.markdown +52 -0
- data/docs/images/AX.png +0 -0
- data/docs/images/all_the_buttons.jpg +0 -0
- data/docs/images/ui_hierarchy.dot +34 -0
- data/docs/images/ui_hierarchy.png +0 -0
- data/ext/key_coder/extconf.rb +6 -0
- data/ext/key_coder/key_coder.m +77 -0
- data/lib/ax_elements/accessibility/enumerators.rb +104 -0
- data/lib/ax_elements/accessibility/language.rb +347 -0
- data/lib/ax_elements/accessibility/qualifier.rb +73 -0
- data/lib/ax_elements/accessibility.rb +164 -0
- data/lib/ax_elements/core.rb +541 -0
- data/lib/ax_elements/element.rb +593 -0
- data/lib/ax_elements/elements/application.rb +88 -0
- data/lib/ax_elements/elements/button.rb +18 -0
- data/lib/ax_elements/elements/radio_button.rb +18 -0
- data/lib/ax_elements/elements/row.rb +30 -0
- data/lib/ax_elements/elements/static_text.rb +17 -0
- data/lib/ax_elements/elements/systemwide.rb +46 -0
- data/lib/ax_elements/inspector.rb +116 -0
- data/lib/ax_elements/macruby_extensions.rb +255 -0
- data/lib/ax_elements/notification.rb +37 -0
- data/lib/ax_elements/version.rb +9 -0
- data/lib/ax_elements.rb +30 -0
- data/lib/minitest/ax_elements.rb +19 -0
- data/lib/mouse.rb +185 -0
- data/lib/rspec/expectations/ax_elements.rb +15 -0
- data/test/elements/test_application.rb +72 -0
- data/test/elements/test_row.rb +27 -0
- data/test/elements/test_systemwide.rb +38 -0
- data/test/helper.rb +119 -0
- data/test/test_accessibility.rb +127 -0
- data/test/test_blankness.rb +26 -0
- data/test/test_core.rb +448 -0
- data/test/test_element.rb +939 -0
- data/test/test_enumerators.rb +81 -0
- data/test/test_inspector.rb +121 -0
- data/test/test_language.rb +157 -0
- data/test/test_macruby_extensions.rb +303 -0
- data/test/test_mouse.rb +5 -0
- data/test/test_search_semantics.rb +143 -0
- metadata +219 -0
@@ -0,0 +1,340 @@
|
|
1
|
+
# Actions (a.k.a. The Fun Part)
|
2
|
+
|
3
|
+
Actions are how you drive the user interface. They can be anything
|
4
|
+
from showing a menu to dragging a window, but are often simple things
|
5
|
+
like pressing a button. Most actions are subtly different cases
|
6
|
+
of the same thing, but it is important to understand the differences
|
7
|
+
and when to use different actions.
|
8
|
+
|
9
|
+
There are three types of actions, those provided by the accessibility
|
10
|
+
API as direct calls, those provided by accessibility APIs as keyboard
|
11
|
+
simulation, and those that are provided by the CGEvents API. In most
|
12
|
+
cases you can use at least two types of actions to achieve the desired
|
13
|
+
effect and each type of action has its own pros and cons to consider.
|
14
|
+
|
15
|
+
## Interface
|
16
|
+
|
17
|
+
Though there are three different APIs under the hood, they are exposed
|
18
|
+
through a common interface via the DSL methods in
|
19
|
+
{Accessibility::Language}.
|
20
|
+
|
21
|
+
app = Accessibility.application_with_name 'Terminal'
|
22
|
+
press app.main_window.minimize_button
|
23
|
+
type "\\CMD+M", app
|
24
|
+
click app.main_window.minimize_button
|
25
|
+
|
26
|
+
The above example demonstrates performing actions using the DSL
|
27
|
+
methods. The DSL methods are always some kind of action, such as
|
28
|
+
{Accessibility::Language#drag_mouse_to drag\_mouse\_to}, or
|
29
|
+
{Accessibility::Language#set_focus set\_focus} to name two.
|
30
|
+
|
31
|
+
The point of the DSL is to put the actions at the front of statements
|
32
|
+
in order to separate verbs from nouns. In this way, scripts should
|
33
|
+
resemble instructions that you might give for reproduction steps in a
|
34
|
+
bug report---they should read like they were instructions for
|
35
|
+
a person. This style should make it easier to think about writing
|
36
|
+
scripts and it should also give better hints at when to refactor
|
37
|
+
code into helper methods like `log_in_as:with_password:`.
|
38
|
+
|
39
|
+
If you tried out the code snippet above you might notice the
|
40
|
+
difference between the APIs (you also might notice that the second
|
41
|
+
command does not work, it is a work in progress :(). In the first
|
42
|
+
case, `press` was using the actions provided by the accessibility APIs
|
43
|
+
to call the press action on the minimize button for the window, in the
|
44
|
+
second case we used `type` to type in the hot key for minimizing a
|
45
|
+
window, and in the third case we used `click` from the CGEvents APIs
|
46
|
+
to actually move the mouse cursor to the button and generate a click
|
47
|
+
event.
|
48
|
+
|
49
|
+
## Accessibility Actions
|
50
|
+
|
51
|
+
Actions like `press` use the accessibility APIs and show up in the
|
52
|
+
list of actions returned by {AX::Element#actions}. These actions will
|
53
|
+
also show up in the Accessibility Inspector in the Actions section.
|
54
|
+
|
55
|
+
You may have noticed that {Accessibility::Language#press press}
|
56
|
+
shows up as `AXPress` for buttons when you view them with the
|
57
|
+
Accessibility Inspector or by calling {AX::Element#actions}. This is
|
58
|
+
because we do name translation with accessibility actions just like
|
59
|
+
when accessing attributes as shown in the
|
60
|
+
{file:docs/Inspecting.markdown Inspection tutorial}.
|
61
|
+
|
62
|
+
Accessibility actions are the simplest and fastest way to trigger
|
63
|
+
actions. They use the accessibility APIs to talk directly to the
|
64
|
+
application to call the same methods that would be called when
|
65
|
+
you interact with the application with the mouse or keyboard, but
|
66
|
+
without having to use the mouse or keyboard.
|
67
|
+
|
68
|
+
## Setting Attributes
|
69
|
+
|
70
|
+
Though it is not really an action, some attributes can have their
|
71
|
+
values changed through the accessibility APIs. The
|
72
|
+
{Accessibility::Language} module has two methods, `set` and
|
73
|
+
`set_focus`, which allow you to change the value for certain
|
74
|
+
attributes.
|
75
|
+
|
76
|
+
In the Accessibility Inspector, a writable attribute will have `(W)`
|
77
|
+
next to the attribute name, and the programmatic way to tell if an
|
78
|
+
attribute is writable is to call {AX::Element#attribute_writable?} and
|
79
|
+
pass the name of the attribute as a parameter:
|
80
|
+
|
81
|
+
app.main_window.attribute_writable? :size
|
82
|
+
app.main_window.attribute_writable? :title
|
83
|
+
|
84
|
+
You can only set an attribute if it is writable. When you have an
|
85
|
+
attribute that is writable, and you would like to change it, then you
|
86
|
+
simply need to call `set`:
|
87
|
+
|
88
|
+
set app.main_window, size: [500, 500].to_size
|
89
|
+
|
90
|
+
The first parameter is the element, and the second parameter is a
|
91
|
+
key-value pair with the attribute as the key and the value as the
|
92
|
+
new value.
|
93
|
+
|
94
|
+
`set_focus` is just syntactic sugar for `set` where the key-value is
|
95
|
+
set for you and the only parameter you provide is the element you want
|
96
|
+
to set focus to:
|
97
|
+
|
98
|
+
set_focus app.main_window.text_field
|
99
|
+
|
100
|
+
Though, `set` itself has a special case. The second most frequently
|
101
|
+
changed attribute for a element is the `value`, it is used almost as
|
102
|
+
much as `focused`. For this case, you can pass anything that is not a
|
103
|
+
key-value pair to `set` and it will be assumed that you want to change
|
104
|
+
the value of the `value` attribute:
|
105
|
+
|
106
|
+
set app.main_window.text_field, 'Mark Rada'
|
107
|
+
|
108
|
+
Another important detail that you might be curious about is that we
|
109
|
+
called `to_size` on the first snippet showing how to set an
|
110
|
+
attribute. The developers of MacRuby have added a feature that allows
|
111
|
+
you to pass an array as an argument when you would normally be
|
112
|
+
expected to pass a structure such as a `CGPoint`. Since AXElements
|
113
|
+
APIs inherently need to handle any type of object, it is not sane to try
|
114
|
+
and do the same type of run time type analysis. In its place, I have
|
115
|
+
provided some convenience methods for {NSArray arrays} to let you
|
116
|
+
quickly transform them to the appropriate type of object. In my
|
117
|
+
opinion, you should just use the proper type of object in the first
|
118
|
+
place and avoid the overhead.
|
119
|
+
|
120
|
+
### You Want To Use Actions
|
121
|
+
|
122
|
+
The advantage to using accessibility actions is that they are usually
|
123
|
+
much easier to use than the other APIs. Accessibility actions are
|
124
|
+
usually synchronous and so your scripts will not need to worry about
|
125
|
+
waiting for animations to complete. The only complication is that
|
126
|
+
sometimes an action is not synchronous and there is no good way to
|
127
|
+
tell if an action is synchronous without looking at the underlying
|
128
|
+
code.
|
129
|
+
|
130
|
+
### Asynchronous Problems
|
131
|
+
|
132
|
+
Problems with synchronous behaviour will occur with all the types of
|
133
|
+
actions, but less often with accessibility actions, so you should
|
134
|
+
always try thing out first in the console to make sure. A good rule of
|
135
|
+
thumb is that anything that looks like it would take a noticeable
|
136
|
+
amount of time, and is not an animation, will probably be
|
137
|
+
asynchronous. An example of this would be having to load data from a
|
138
|
+
database (read: separate process, across a network, etc.) such as in
|
139
|
+
the Marketcircle server admin apps or the Billings Pro preferences.
|
140
|
+
|
141
|
+
Asynchronous behaviour can often be worked around by using
|
142
|
+
accessibility {file:docs/Notifications.markdown notifications} or by
|
143
|
+
simply calling
|
144
|
+
[`sleep`](http://rdoc.info/stdlib/core/1.9.2/Kernel#sleep-instance_method).
|
145
|
+
Notifications have some interesting caveats that are covered in their
|
146
|
+
own tutorial and so it is much easier to `sleep`.
|
147
|
+
|
148
|
+
However, using `sleep` to solve the problem of asynchronous waiting is
|
149
|
+
like using a steak knife for brain surgery. Unless you can control the
|
150
|
+
environment when the script is running, you will need to sleep for
|
151
|
+
longer periods of time than you really need to; even when you have
|
152
|
+
quite a bit of control you might still have the occasional instance
|
153
|
+
where a database fetch takes longer than expected. I would suggest you
|
154
|
+
use notifications when you can, and sleep when you cannot.
|
155
|
+
|
156
|
+
## CGEvents Actions
|
157
|
+
|
158
|
+
CGEvents based actions never directly use the accessibility APIs and
|
159
|
+
are implemented at a different level of the GUI stack in OS X (I
|
160
|
+
assume). An action that uses CGEvents will actually move the mouse
|
161
|
+
cursor around and simulate mouse input at a relatively low level of
|
162
|
+
abstraction. However, accessibility information is still used to find
|
163
|
+
the point on the screen to move to and click. These types of actions
|
164
|
+
are more realistic, more awesome looking, and also more difficult to
|
165
|
+
write.
|
166
|
+
|
167
|
+
The difficulty in writing the scripts comes from the fact that it does
|
168
|
+
not directly communicate with the accessibility APIs. This implicitly
|
169
|
+
means that all CGEvents actions are asynchronous, which is the only
|
170
|
+
non-trivial complication with using CGEvents APIs.
|
171
|
+
|
172
|
+
### CGEvents Goes Where Accessibility Cannot
|
173
|
+
|
174
|
+
The benefit of actually moving the mouse will often outweigh the
|
175
|
+
downside here. Moving the mouse cursor and generating click events
|
176
|
+
allows things like dragging and scrolling which cannot be done using
|
177
|
+
the accessibility APIs.
|
178
|
+
|
179
|
+
As mentioned earlier, CGEvents doesn't talk to applications in the
|
180
|
+
same way that accessibility APIs do, which means that you can use
|
181
|
+
CGEvents actions on elements that do not fully support
|
182
|
+
accessibility. For instance, text might appear on the UI and have a
|
183
|
+
`click`-able link but may not provide an equivalent action to clicking
|
184
|
+
on the link; using CGEvents APIs you will only need to move to the
|
185
|
+
position of the text and then generate a click event.
|
186
|
+
|
187
|
+
### Realism
|
188
|
+
|
189
|
+
Since CGEvents actually manipulates the mouse cursor, any script that
|
190
|
+
uses the APIs will be more realistic than their equivalent
|
191
|
+
accessibility actions. This can make a difference if you are testing
|
192
|
+
an app and you are expecting other side effects to occur. For instance,
|
193
|
+
what if you implemented a sort of UI element that requires the mouse
|
194
|
+
cursor to be near the element for an action to trigger, such as an
|
195
|
+
expanding drawer or a folder in the finder. Depending on what you are
|
196
|
+
using AXElements for, this may or may not be important to you.
|
197
|
+
|
198
|
+
### Underpinnings
|
199
|
+
|
200
|
+
An important thing to note is that AXElements always works with
|
201
|
+
flipped co-ordinates. The origin, `(0, 0)` is always located in the
|
202
|
+
top left corner of the main display. Displays to the left have a
|
203
|
+
negative `x` co-ordinate and displays above the main display will have
|
204
|
+
negative `y` co-ordinates.
|
205
|
+
|
206
|
+
Though the CGEvents APIs are exposed to AXElements through the
|
207
|
+
{Accessibility::Language Language} module, there is another thin
|
208
|
+
layer between AXElements and CGEvents that abstracts everything that
|
209
|
+
AXElements uses.
|
210
|
+
|
211
|
+
The between layer is the {Mouse} module, and it is responsible for
|
212
|
+
generating events and animating the mouse movements on screen. The API
|
213
|
+
that it makes available to AXElements is more fine grained than what
|
214
|
+
AXElements makes available to you. There is a lot of room for the
|
215
|
+
{Mouse} module to grow and add new features, some of which are noted
|
216
|
+
as `@todo` items in the documentation. It may also be necessary in
|
217
|
+
the future to make available from AXElements certain options that are
|
218
|
+
normally hidden in {Mouse}.
|
219
|
+
|
220
|
+
### Where The Mouse Moves
|
221
|
+
|
222
|
+
Consider the following:
|
223
|
+
|
224
|
+
move_mouse_to app.main_window.close_button
|
225
|
+
|
226
|
+
If you try out that code snippet then you will notice that the mouse
|
227
|
+
cursor moves to the button, specifically the center of the button. If
|
228
|
+
you use the method to move to another object, such as a button, you
|
229
|
+
will again notice that it moved to the center point of the element.
|
230
|
+
|
231
|
+
You can call `move_mouse_to` and give it a UI element, but you could
|
232
|
+
also be more exact and give it a CGPoint or even an array with two
|
233
|
+
numbers. In fact, you could pass just any object as long it implements
|
234
|
+
a method named `to_point` which returns a CGPoint object. If you were
|
235
|
+
to look at the code for {Accessibility::Language#move_mouse_to} you
|
236
|
+
would see that all it does is call `#to_point` on the argument and
|
237
|
+
then pass the returned value to the {Mouse} module to actually do the
|
238
|
+
work. You could also look at {NSArray#to_point} and find out that it just
|
239
|
+
returns a new CGPoint using the first two objects in the array, and
|
240
|
+
`CGPoint#to_point` just returns itself (you can't see the
|
241
|
+
documentation for the method because of limitations in
|
242
|
+
YARD). Similarly, {AX::Element} implements
|
243
|
+
{AX::Element#to_point to_point} which not only gets the position for
|
244
|
+
the element, but also gets the size and then calculates the center
|
245
|
+
point for the element. This is important because you probably never
|
246
|
+
want the mouse cursor moving to the top left of the UI element, but
|
247
|
+
maybe you don't want the cursor moving to the center either.
|
248
|
+
|
249
|
+
If you want to have the mouse move to a location over a UI element
|
250
|
+
that is not the center point, then you will need to implement
|
251
|
+
`to_point` for the appropriate class. Just remember to follow the
|
252
|
+
{file:docs/NewBehaviour.markdown rules} for choosing the proper
|
253
|
+
superclasss. For instance, on Snow Leopard you could resize a window
|
254
|
+
by clicking and dragging an `AX::GrowArea` object in the lower right
|
255
|
+
corner of a window; moving to the center of the element may not
|
256
|
+
actually allow the mouse to respond and so you would have to implement
|
257
|
+
`to_point` to move closer to the bottom right of the window
|
258
|
+
(__NOTE__: I don't know if that is actually true for `AX::GrowArea`,
|
259
|
+
it was just meant as an example).
|
260
|
+
|
261
|
+
### Dragging
|
262
|
+
|
263
|
+
Sometimes you just need to be able to click and drag an element. To do
|
264
|
+
this you can only use the CGEvents APIs. Fortunately, click and drag
|
265
|
+
is relatively painless with AXElements, the simplest example would be
|
266
|
+
something like this:
|
267
|
+
|
268
|
+
# move to the mouse to the starting point
|
269
|
+
move_mouse_to app.main_window.title_ui_element
|
270
|
+
|
271
|
+
# start the click and drag event
|
272
|
+
drag_mouse_to [0, 0]
|
273
|
+
|
274
|
+
Pretty cool, eh? The general pattern is to move the mouse to the
|
275
|
+
starting point, and then to call `drag_mouse_to` to start the click
|
276
|
+
and drag events. In the example, I gave co-ordinates using an array
|
277
|
+
with two numbers but you can pass a CGPoint or anything else that
|
278
|
+
responds to `to_point` just like with the other CGEvents actions.
|
279
|
+
|
280
|
+
## Keyboard Actions
|
281
|
+
|
282
|
+
The final type of action is the keyboard action. Keyboard actions
|
283
|
+
live in a world between the accessibility APIs and CGEvent APIs; that
|
284
|
+
is, Apple has already done the work of unifying them for me and in a
|
285
|
+
way that I would not have been able to do myself.
|
286
|
+
|
287
|
+
The keyboard action is a single method,
|
288
|
+
{Accessibility::Language#type}, that requires one parameter and
|
289
|
+
takes an optional second parameter. The first parameter is simply a
|
290
|
+
string that includes which characters you wish to have typed out and
|
291
|
+
the second parameter is the UI element for the application where you
|
292
|
+
would like to send the keyboard events. Since the second parameter is
|
293
|
+
optional, the currently focused app will receive the events if the
|
294
|
+
parameter is not used.
|
295
|
+
|
296
|
+
# to a specific app
|
297
|
+
app = Accessibility.application_with_name 'Terminal'
|
298
|
+
type "Hello, world!", app
|
299
|
+
|
300
|
+
# to the focused app
|
301
|
+
type "Hello, world!"
|
302
|
+
|
303
|
+
The string that you pass can have some special escape sequences to
|
304
|
+
indicate that you want to press command characters or a combination of
|
305
|
+
keys at once for a hot key. Details on this are contained in their own
|
306
|
+
tutorial, the
|
307
|
+
{file:docs/KeyboardEvents.markdown Keyboard Events Tutorial}.
|
308
|
+
|
309
|
+
## Mixing In
|
310
|
+
|
311
|
+
By now you will have noticed that all the actions have been defined in
|
312
|
+
the {Accessibility::Language} name space, which is just a simple
|
313
|
+
module. Though, you have been able to use the methods anywhere and in
|
314
|
+
any context.
|
315
|
+
|
316
|
+
When you load AXElements, by using `require 'ax_elements'`, not only
|
317
|
+
will all the code be loaded, but the extra step of mixing
|
318
|
+
{Accessibility::Language} into the top level name space will also be
|
319
|
+
done for you. The only way to override this behaviour is to load the
|
320
|
+
components of AXElements yourself, the details on how to do this are
|
321
|
+
left as an exercise to interested parties. :P
|
322
|
+
|
323
|
+
## A Note About Caching
|
324
|
+
|
325
|
+
You need to be careful when you cache UI element objects. Every time
|
326
|
+
that you trigger an action you are intrinsically changing the state of
|
327
|
+
an application. State changes will often cause new UI elements to be
|
328
|
+
created, recreated, or removed.
|
329
|
+
|
330
|
+
For example consider pressing the close button for a window; in this
|
331
|
+
case, an entire window and all of its children will disappear and
|
332
|
+
become invalid objects. Another case might be pressing the `+` button
|
333
|
+
for a table; in this case you have created a new row for the table and
|
334
|
+
any cache of the existing rows for the table will not include the new
|
335
|
+
element.
|
336
|
+
|
337
|
+
The real problem with caching is with the invalid objects. An invalid
|
338
|
+
object is poisonous to the MacRuby run time. If you try to access an
|
339
|
+
attribute or trigger a search from an invalid then you will cause
|
340
|
+
MacRuby to crash.
|
@@ -0,0 +1,326 @@
|
|
1
|
+
# Debugging
|
2
|
+
|
3
|
+
This document includes instructions on using AXElements' built in
|
4
|
+
tools to help you debug issues with your scripts, or in cases where
|
5
|
+
you might find a bug in AXElements itself or MacRuby.
|
6
|
+
|
7
|
+
## Trees
|
8
|
+
|
9
|
+
Sometimes you need to see the big picture, the whole UI tree at
|
10
|
+
once or at least be able to see the root of the hierarchy from where
|
11
|
+
you are. For these troubling cases AXElements provides a few tools.
|
12
|
+
|
13
|
+
### Text Tree
|
14
|
+
|
15
|
+
Printing a text tree is similar to how a UI dump works with
|
16
|
+
accessibility on iOS. AXElements does improve a bit on its iOS
|
17
|
+
equivalent. Simply using the Accessibility Inspector does not give you
|
18
|
+
a good picture of the whole tree and with complicated structures it is
|
19
|
+
easy to make mistakes navigating the UI tree.
|
20
|
+
|
21
|
+
The text tree comes formatted for you, and you can simply print it out
|
22
|
+
to the console. The output uses indentation to indicate how far down
|
23
|
+
the tree each element is, and the first element up the tree with one
|
24
|
+
indentation level less will always be the parent for an element.
|
25
|
+
|
26
|
+
Printing out a text tree in the console is very easy. First you generate
|
27
|
+
the text dump and then you print it out:
|
28
|
+
|
29
|
+
puts Accessibility.dump(app)
|
30
|
+
|
31
|
+
This method is useful when you are having difficulties with
|
32
|
+
search. Sometimes when searching you will end up finding the wrong
|
33
|
+
element because your search query was ambiguous and you didn't
|
34
|
+
know at the time. Printing out a dump can often be all the help you
|
35
|
+
need in order to identify your problem.
|
36
|
+
|
37
|
+
However, if the standard `#inspect` isn't doing it for you, you can
|
38
|
+
perform your own inspection every element in a tree yourself using the
|
39
|
+
built in enumerators. Searches use the {Accessibility::BFEnumerator},
|
40
|
+
but the text tree dump method uses {Accessibility::DFEnumerator}.
|
41
|
+
|
42
|
+
### Dot Graph
|
43
|
+
|
44
|
+
__NOTE__: This feature isn't actually done yet. Coming soon, I promise.
|
45
|
+
|
46
|
+
For super fancy text trees, AXElements can generate dot graphs for
|
47
|
+
consumption by [Graphviz](http://www.graphviz.org/). In this case, you
|
48
|
+
want to call {Accessibility.graph} and pass the root of the tree you
|
49
|
+
want to have turned into a dot graph; you will get a string back that
|
50
|
+
you will then need to give to Graphviz in order to generate the visual
|
51
|
+
graph.
|
52
|
+
|
53
|
+
### Text Tree Won't Print?
|
54
|
+
|
55
|
+
AXElements isn't perfect and it is possible that an edge case slipped
|
56
|
+
between the cracks. However, it's also possible that the problem is
|
57
|
+
actually how accessibility information is being vended out.
|
58
|
+
|
59
|
+
As an example, Radar #10040865 is a ticket that I filed with Apple
|
60
|
+
where they broke accessibility with search fields. The effect to
|
61
|
+
AXElements was that you could not search through the menu bar causing
|
62
|
+
a text dump to fail in a very difficult to trace manner.
|
63
|
+
|
64
|
+
In these cases you will need to use your deductive reasoning to figure
|
65
|
+
out where the problem is coming from. Fortunately, I have provided
|
66
|
+
some tools to help you along the way.
|
67
|
+
|
68
|
+
## All The Way Up
|
69
|
+
|
70
|
+
Sometimes you don't need a whole tree to be printed out. Sometimes
|
71
|
+
just seeing the path from an element to the top level element is
|
72
|
+
enough. {Accessibility.path} is your friend in this case, it will
|
73
|
+
provide you with an array of UI element objects, each successive
|
74
|
+
element will be the parent of the previous element. This is almost
|
75
|
+
like the view that the Accessibility Inspector provides.
|
76
|
+
|
77
|
+
## Custom Exceptions
|
78
|
+
|
79
|
+
AXElements provides some customized exceptions in the OO layer that
|
80
|
+
should help give you much better hints at what went wrong when you
|
81
|
+
have a problem.
|
82
|
+
|
83
|
+
Custom exceptions have been created to help identify the point of
|
84
|
+
failure that would have caused a more cryptic exception to have been
|
85
|
+
raised instead. These custom exceptions also capture more metadata
|
86
|
+
about the problem that occurred which should give good hints as to what
|
87
|
+
went wrong.
|
88
|
+
|
89
|
+
### Search Failures
|
90
|
+
|
91
|
+
An {AX::Element::SearchFailure `SearchFailure`} will occur when you
|
92
|
+
perform an implicit search that fails to find anything.
|
93
|
+
|
94
|
+
In cases where implicit searches are chained, which happens frequently
|
95
|
+
with deep UI hierarchies, if one of the searches were to fail then you
|
96
|
+
would receive a `NoMethodError` about something having
|
97
|
+
failed. Sometimes the failure would happen because the search returned
|
98
|
+
`nil`, and of course `nil` would not respond to another search, though
|
99
|
+
this problem was easy to identify if you are familiar with the
|
100
|
+
AXElements source code; in other cases the failure would occur
|
101
|
+
somewhere in the search {Accessibility::Qualifier qualifier} or in the
|
102
|
+
{Accessibility::BFEnumerator enumerator}, and it was not always clear why.
|
103
|
+
|
104
|
+
The other feature of a search failure is that the exception message
|
105
|
+
will include an element back trace using {Accessibility.path}. This is
|
106
|
+
meant to give a hint about why the search failed.
|
107
|
+
|
108
|
+
### Attribute Not Writable
|
109
|
+
|
110
|
+
You will receive {AX::Element::ReadOnlyAttribute `ReadOnlyAttribute`}
|
111
|
+
exceptions only when you try to set an attribute that is not
|
112
|
+
writable. Again, this was originally designed to more easily identify the
|
113
|
+
point of failure when you try to write to an attribute that you should
|
114
|
+
not write to.
|
115
|
+
|
116
|
+
Specifically, `set_focus` is called by methods internally by
|
117
|
+
AXElements and at one time it was causing some problems when elements
|
118
|
+
were unexpectedly not allowing their `focused` attribute to be
|
119
|
+
written.
|
120
|
+
|
121
|
+
### Attribute Not Found
|
122
|
+
|
123
|
+
A very simple fail safe that AXElements uses is the
|
124
|
+
{AX::Element::LookupFailure `LookupFailure`} exception which will be
|
125
|
+
raised when you try to explicitly access an attribute which does not
|
126
|
+
exist, or at least does not exist for the particular element that you
|
127
|
+
are trying to access.
|
128
|
+
|
129
|
+
## Not So Custom Exceptions
|
130
|
+
|
131
|
+
Sometimes it is possible that the back trace for other exceptions can
|
132
|
+
get lost. This may be a result of
|
133
|
+
[MacRuby Ticket #1369](http://www.macruby.org/trac/ticket/1369), but
|
134
|
+
it might also be because of
|
135
|
+
[MacRuby Ticket #1320](http://www.macruby.org/trac/ticket/1320) or
|
136
|
+
some other freedom patch
|
137
|
+
that AXElements or ActiveSupport adds to the run time. I have not been
|
138
|
+
able to create a reduction of the problem yet.
|
139
|
+
|
140
|
+
The real problem is that loss of back trace happens for multiple
|
141
|
+
exception classes. The [work around](https://gist.github.com/1107314)
|
142
|
+
for this case is copied to `lib/ax_elements/macruby_extensions.rb`,
|
143
|
+
but has been commented out since it causes some regression tests to
|
144
|
+
fail. If you do not get a back trace with an error then you will need
|
145
|
+
to uncomment the freedom patches or copy them to your own script.
|
146
|
+
|
147
|
+
## Disabling Compiled Code
|
148
|
+
|
149
|
+
Back traces can be lost for reasons other than a bug. When using
|
150
|
+
compiled MacRuby code, you cannot get a Ruby level back trace in case
|
151
|
+
of an error. This feature is on the road map for MacRuby, but I am not
|
152
|
+
sure when it will be done.
|
153
|
+
|
154
|
+
In the mean time, if you suspect that the portion of a back trace that
|
155
|
+
would come from a compiled file is the problem, then you can disable
|
156
|
+
loading compiled files which will force MacRuby to load source ruby
|
157
|
+
files instead. You can disable loading compiled files by setting the
|
158
|
+
`VM_DISABLE_RBO` environment variable before running a script. You can
|
159
|
+
disable loading for a single session like so:
|
160
|
+
|
161
|
+
VM_DISABLE_RBO=1 macruby my_script.rb
|
162
|
+
|
163
|
+
Other debugging options are also available from MacRuby itself. You
|
164
|
+
should check out [Hacking.rdoc](https://github.com/MacRuby/MacRuby/blob/master/HACKING.rdoc)
|
165
|
+
in the MacRuby source repository for more details.
|
166
|
+
|
167
|
+
## Logging
|
168
|
+
|
169
|
+
The core level of AXElements has logging in every case that an error
|
170
|
+
code is returned. Though, it can show false positives because of
|
171
|
+
hiccups in the accessibility API or implementation that an app
|
172
|
+
provides; so it turned off by default. This feature is also going away
|
173
|
+
in favour of more intelligent error handling in the core wrapper.
|
174
|
+
|
175
|
+
When weird bugs are occuring, possibly even crashing MacRuby, you
|
176
|
+
should try turning on the logs and then trying the script again. You
|
177
|
+
might see some tell tale logs printed out right before the crash. You
|
178
|
+
can turn on logging right after you load AXElements, like so:
|
179
|
+
|
180
|
+
require 'ax_elements'
|
181
|
+
Accessibility.log.level = Logger::DEBUG
|
182
|
+
|
183
|
+
The standard log levels are available, the full set is available
|
184
|
+
[here](http://rdoc.info/stdlib/logger/1.9.2/Logger/Severity). `Logger::DEBUG`
|
185
|
+
will turn on all logs.
|
186
|
+
|
187
|
+
## MacRuby Seems Slow
|
188
|
+
|
189
|
+
There are a few things that can cause MacRuby to be slow. At boot time
|
190
|
+
there are a number of factors, which I will cover, and at run time
|
191
|
+
there is really only one culprit.
|
192
|
+
|
193
|
+
### Long Load Times
|
194
|
+
|
195
|
+
When using certain gems, or when you have many gems installed, you
|
196
|
+
will notice that the load time for your scripts is very
|
197
|
+
long---possibly more than 10 seconds. There are many reasons why this
|
198
|
+
happens, some of which we can fix ourselves and some of which you will
|
199
|
+
have to wait for the MacRuby developers to fix.
|
200
|
+
|
201
|
+
#### Huge Literal Collections
|
202
|
+
|
203
|
+
Some gems contain source code with
|
204
|
+
[unbelievably large literal collections](https://github.com/sporkmonger/addressable/blob/master/lib/addressable/idna/pure.rb#L318),
|
205
|
+
such as the `addressable` gem. This is a problem for MacRuby for two
|
206
|
+
reasons.
|
207
|
+
|
208
|
+
First, it requires several thousand allocations at once. Most of
|
209
|
+
MacRuby's performance issues come from code that allocates too much,
|
210
|
+
and large collections can allocate several thousand objects all at
|
211
|
+
once.
|
212
|
+
|
213
|
+
The second problem is that MacRuby normally will try to JIT the code,
|
214
|
+
and the LLVM chokes on things this large. Some work has been done to
|
215
|
+
break the function up into smaller pieces (it used to take over 2
|
216
|
+
minutes to load the `addressable` gem), but it can still take a while
|
217
|
+
for MacRuby and the LLVM to work through the code
|
218
|
+
|
219
|
+
As it turns out, JIT isn't that great for short lived processes that
|
220
|
+
need to start up over and over again. In fact, JIT mode for MacRuby
|
221
|
+
was meant more for debugging.
|
222
|
+
|
223
|
+
The work around to this situation will have to come from upstream gem
|
224
|
+
developers and MacRuby itself. In the mean time, compiling these gems
|
225
|
+
will usually make them load _significantly_ faster. To compile gems,
|
226
|
+
you can install the
|
227
|
+
[`rubygems-compile`](https://github.com/ferrous26/rubygems-compile)
|
228
|
+
plug-in for rubygems. Follow the instructions from the plug-ins `README`
|
229
|
+
to learn how to use it and to know which version to install.
|
230
|
+
|
231
|
+
#### Complex Metaprogramming
|
232
|
+
|
233
|
+
Another problem that can cause long load times is complex
|
234
|
+
metaprogramming. Gems such as `rspec` do a lot of weird stuff at boot
|
235
|
+
that causes the MacRuby optimizer to do a lot of work. `rspec` alone
|
236
|
+
can add nearly 10 seconds to boot time. In this case you can tell the
|
237
|
+
optimizer to not try so hard; this will result in slower run time
|
238
|
+
performance, but it is likely worth the trade off in the case of
|
239
|
+
`rspec` (unless you compile).
|
240
|
+
|
241
|
+
You can set the optimization level for MacRuby just as you would
|
242
|
+
disable loading compiled code:
|
243
|
+
|
244
|
+
# set the level to a number between 0 and 3, 3 is the highest
|
245
|
+
VM_OPT_LEVEL=1 macruby my_script.rb
|
246
|
+
|
247
|
+
#### Large Code Bases
|
248
|
+
|
249
|
+
Large code bases taking a long time to load is not really an avoidable
|
250
|
+
situation---it happens with every project. Once again, JIT and
|
251
|
+
optimizer passes take up a lot of the load time.
|
252
|
+
|
253
|
+
In this case, it is best to compile your code (and test the compiled
|
254
|
+
version) in order to speed up boot time. You can combine compiled code
|
255
|
+
and still turn off optimization passes to get even better boot times,
|
256
|
+
but I am not sure it is worth the trade off at that point.
|
257
|
+
|
258
|
+
#### Rubygems
|
259
|
+
|
260
|
+
Rubygems suffers from a lot of technical debt. The process of
|
261
|
+
activating a gem incurs so many allocations that with as few as 25
|
262
|
+
installed gems can add an extra 2 seconds to your boot time. What is
|
263
|
+
worse, the performance degrades exponentially as you install more
|
264
|
+
gems.
|
265
|
+
|
266
|
+
The only fix for this is to cleanup and fix rubygems. Fortunately this
|
267
|
+
has been underway since the rubygems 1.4; the downside is that MacRuby
|
268
|
+
has customizations to rubygems that prevent users from upgrading
|
269
|
+
themselves. We need to wait for new MacRuby releases to bundle new
|
270
|
+
rubygems versions in order to fix this issue.
|
271
|
+
|
272
|
+
### Slow Runtime Performance
|
273
|
+
|
274
|
+
In my experience, slow runtime performance in MacRuby is almost always
|
275
|
+
the result of many allocations. If your code is runinng abnormally
|
276
|
+
slow then it is likely that you are allocating a lot of memory without
|
277
|
+
realizing it, and you should compare performance to CRuby if it is
|
278
|
+
important (and if it is possible to run the code on CRuby).
|
279
|
+
|
280
|
+
Remember that things like literal strings have to be copied every time
|
281
|
+
the line of code they are on is run, whereas immutable things like
|
282
|
+
symbols do not have to be copied. At the same time, if you pass a
|
283
|
+
symbol to a method that will coerce the symbol to a string then you
|
284
|
+
haven't saved an allocation.
|
285
|
+
|
286
|
+
Try to use in-place mutations when it is safe. An example would be
|
287
|
+
when you have to perform multiple changes to an object in a single
|
288
|
+
method, you only have to create a new copy the first time and then use
|
289
|
+
the same copy for all the other changes. Example code would look like this:
|
290
|
+
|
291
|
+
def transform string
|
292
|
+
new_string = string.gsub /pie/, 'cake'
|
293
|
+
new_string.gsub! /hate/, 'love'
|
294
|
+
new_string.upcase!
|
295
|
+
new_string
|
296
|
+
end
|
297
|
+
|
298
|
+
Remember that built-in in-place methods tend to return `nil` if they
|
299
|
+
don't make any changes, which means you need to explicitly return the
|
300
|
+
new object at the end of the method. There are still many other easily
|
301
|
+
avoidable cases where you could end up allocating a lot of memory
|
302
|
+
which are not covered. If it is important you willl just have to
|
303
|
+
analyze your code.
|
304
|
+
|
305
|
+
Sometimes allocating a lot of memory is not avoidable; running RSpec
|
306
|
+
would be an example of this. In these cases you just have to bite the
|
307
|
+
bullet.
|
308
|
+
|
309
|
+
## Don't Be Afraid To Log Bugs
|
310
|
+
|
311
|
+
Or look at the AXElements source code for that matter. The source is
|
312
|
+
well documented and hopefully not too clever, so it shouldn't be too
|
313
|
+
hard to figure things out. You can log AXElements bugs on Github where
|
314
|
+
the source is being hosted.
|
315
|
+
|
316
|
+
Though, sometimes the problem will be a MacRuby problem and the best
|
317
|
+
way to get it fixed is to
|
318
|
+
[log a bug](http://www.macruby.org/trac/). You will need to create an
|
319
|
+
account with them to log bugs.
|
320
|
+
|
321
|
+
I also recommend that you subscribe to the
|
322
|
+
[MacRuby mailing list](http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel)
|
323
|
+
to keep up to date on MacRuby developments. You can send in questions
|
324
|
+
if you are unsure about a certain behaviour, even potential bugs. It
|
325
|
+
is not a high traffic mailing list so it won't blow up your mailbox
|
326
|
+
when you subscribe.
|