The functions fozzie_difference_join_rs,
fozzie_distance_join_rs,
fozzie_interval_join_rs, fozzie_regex_join_rs,
and fozzie_string_join_rs functions are no longer exported
and their .Rd documentation files have been removed.
Description file updated to add references for ‘fuzzyjoin’ as well
as the Rust crates ‘textdistance’, ‘rapidfuzz’, and ‘interavl’
fozziejoin 0.0.12
Enhanced the testing suite for multithreading validation. The
testing suite is configured to use, at most, 2 threads. To ensure this
is honored, we check that the user space time is less than or equal to
2.5 times the wall clock time. However, when datasets become very small,
these tests can fail sporadically. Tests have been updated to use larger
datasets for better reliability.
Updated the DESCRIPTION file to remove direct references to
tibbles in the Description field to be more generic and
formal.
fozziejoin 0.0.11
Converted relative hyperlinks in the README file to absolute
hyperlinks
Fixed remaining references to the old, archived GitHub repo
Added inst/AUTHORS file to list the authors of dependency Rust
crates in addition to the LICENSE.note
Now that we have started the CRAN submission cycle, precompiled
binaries for Windows will no longer be created. Relevant sections of the
README have been updated.
fozziejoin 0.0.10
Two vignettes added:
General package overview
Benchmarking sample and considerations
Interval join with interval_mode = 'real' now handles a
mix of integer and double inputs correctly.
If by = NULL, the internal common_by
function will now print the columns used in the join.
License information updated to reflect author(s) of all imported
Rust crates. This seems necessary based on a review of other similar
extendr packages.
Reproducible benchmark scripts created using github workflows
Users can now set a global thread count via
options(fozzie.nthread = 4), which will be respected by all
functions with an nthread argument. By default, the package uses the
default from the multithreading Rust library rayon.
Initial CRAN submission
fozziejoin 0.0.9
Distance joins now available.
Semi join type added.
If one of the input dataframes is a tibble, the output
result will now be a tibble.
This is necessary to handle some of the functionalities present in
tibble but not in data.frame.
tibble is now a suggested import.
Interval joins added, with three “interval_mode”’s:
integer: integer-based join types, with behavior
designed to emulate IRanges findOverlaps. Importantly, [1, 2] and [3, 4]
would be considered overlapping in this case.
real: real number joins, where there must be some
continuous overlap between ranges to be considered matching.
auto: behavior determined by the input column
types.
The by function should now better resemble the
fuzzyjoin implementation. Notes have been added to the
internal function signature to acknowledge their contribution.
Performance improvements.
Rust code now uses FxHashMap and FxHashSet universally.
Simplified memory structures for case when only one column is joined
on.
Better code organization in Rust code.
Better error handling.
Most areas of the code now gracefully return an error to R instead
of panicking.
The areas where panics still might happen aren’t known to throw
errors, but I’d still like to properly handle them in the future.
Now using styler to be more style guide compliant.
fozziejoin 0.0.8
Arbitrary vector attributes, such as factor levels and POSIX dates,
should now be supported. See: Issue #6.
Testing utilities updated to validate this change.
Fixed a bug in the nthread argument wherein the
user-specified thread count was ignored and the default global thread
pool settings were always used. See Issue
#7.
Contributor code of conduct added
string distance functions added to their own submodule within the
Rust code. This is to better organize the code as we plan to add other
fuzzy join types (distance, difference, geo, etc.)
fozzie_join functions have been renamed to
fozzie_string_join. This will better describe the function
behavior and allow us to add other join types in the future. See Issue
#9
fozzie_string_full_join now implements full joins as
the union of the left and right fuzzy join. Before this, it was the
cartesian product of left and right datasets.
fozzie_difference_join suite of functions now
available. This allows joining on numeric distance.
fozziejoin 0.0.7
Switched to rapidfuzz crate for supported algorithms,
as they perform better than prior implementations.
README updates
.gitignore updated to remove vendored packages, as is
convention.
fozziejoin 0.0.6
Fixed issue with Jaccard and qgram distance (see issue
#3).
Comparative benchmark vs. fozziejoin updated to check for identical
output (after some light conversions for consistency in column
naming/output object classes).
fozziejoin 0.0.5
Functionality and
performance updates
Joins now properly handle dates and factors
Added convenience function for all directional variants of joins
(fozzie_left_join(), fozzie_inner_join(),
…).
Reverted a change from v0.0.4 wherein speed distance calculation
methods differ by operating system (Windows vs. everything else). The
supposed speed gains were actually flaws in the evaluation. Reverted
back to a single method for all OS’s.
Speedup in OSA algorithm due to more efficient memory handling.
Documentation
README updates:
Installation steps reflect current procedures and reference the
GitHub release for v0.0.5.
Requirements updated as there is now an install from binary option
for Windows which has fewer system requirements.
Removed Todo section. Will use GitHub issues for this sort of thing
moving forward.
Documentation had error in example usage code.
fuzzyjoin was a required import for the
misspellings dataset.
Documentation updated to pass all devtools::check() and
R CMD check checks for the first time.
There are a few examples where code is only lightly adapted from the
textdistance crate implementation. Those scripts now have a
header comment acknowledging the original author.
Preparation for CRAN release
This version is the last before attempting CRAN distribution. A
GitHub “release” has been created with the package build for all
operating systems. CRAN acceptance may require multiple versions.
All tests now force nthread=2 for compliance with CRAN
policies.
fozziejoin 0.0.4
Performance improvements:
Windows build now uses a parallelization method more appropriate for
the OS (rayon’s par_chunks have replaced equivalent
par_iter operations)
Q-gram based edit distances have been sped up by reducing memory
copies.
Scripts for benchmarking have been added.
Project README updated to include some benchmarking results.
fozziejoin 0.0.3
Anti join implemented
Full join implemented
Multikey joins now allowed (e.g. joining on “Name” and “DOB”).
LCS string distance now available. This matches the original R
stringdist behavior.
Can control number of threads using the nthread
parameter.
Jaro-Winkler parameters prefix_weight and
max_prefix parameters added. These are similar to the
bt and p parameters in the
stringdist package, with some differences
(prefix_weight is a set number of characters, not a
proportion).
The jaro method is no longer supported. The default
values for the jw and jaro_winkler methods
simplify into the Jaro case.
Removed case insensitive matching as an immediate project goal.
fozziejoin 0.0.2
Right-hand join functionality implemented.
The parameter distance_col is live. It can be used to
add the string distance of joined fields to the output.
Fixed an issue where left and right joins would replace
NA in R character fields with a string with the string
value “NA”. Tests updated to expect a true NA.
Added explicit checks for NA strings in all Rust
internals that perform fuzzy matches. If one or more values in a pair is
NA, the pair is considered a non-match.
Updated README.
fozziejoin 0.0.1
NEWS.md added
Inner join implemented for all string distance algorithms except
LCS
Most string distance algorithms have been implemented for
inner and left joins. Results were verified
against expectations and with the fuzzyjoin package.
Exceptions:
jarowinkler/jw method requires the
addition of new parameters for p and dt to be
fully customizable. Currently, jaro_winkler defaults to a scaling factor
of 0.1 and a maximum prefix of 4. This is consistent with the default of
the stringdist method.
jaro algorithm does not actually exist in the
stringdist implementation, as it is equivalent to setting
p=0.
LSA algorithm is not implemented yet. There is an
implementation in the Rust code, but it is not correct and the R
user has no way of calling that method.
Project DESCRIPTION file updated
fuzzy_join API call now includes the how
method to specify the join type. inner and
left are the currently supported methods. At least
right, full, and anti are planned
for future releases.