"The social media company will soon stop counting photos and links as part of its 140-character limit for messages, according to a person familiar with the matter." (Source)

Will this library be updated to match these changes?

We've found an odd error trying to use twitter-text in a rails app. It works fine from the irb, and so we are a little confused.

Here's it all good in IRB:

[tansaku@Samuels-MBP:~/Documents/Github/AgileVentures/LocalSupport (131428629_automated_posting_to_social_media_feeds)]$ 
→ bundle exec irb
2.4.2 :001 > require 'twitter-text'
 => true 
2.4.2 :002 > include Twitter::Validation
 => Object 
2.4.2 :003 > parse_tweet "Test Tweet 4 https://www.harrowcn.org.uk/"
 => {:weighted_length=>36, :valid=>true, :permillage=>128, :valid_range_start=>0, :valid_range_end=>40, :display_range_start=>0, :display_range_end=>40} 

here's the full stack trace when we try to run in a test:

  1) TwitterApi Posting to twitter should be able to post 280 chars or less to twitter
     Failure/Error: result = parse_tweet(tweet)
     
     NoMethodError:
       undefined method `[]' for nil:NilClass
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/twitter-6.2.0/lib/twitter/base.rb:128:in `attr_falsey_or_empty?'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/twitter-6.2.0/lib/twitter/base.rb:86:in `block in define_attribute_method'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/method_builder.rb:117:in `call'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/method_builder.rb:117:in `block (3 levels) in create_memoized_method'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/memory.rb:63:in `block (3 levels) in fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/thread_safe-0.3.6/lib/thread_safe/cache.rb:56:in `fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/memory.rb:62:in `block (2 levels) in fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/memory.rb:61:in `block in fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/thread_safe-0.3.6/lib/thread_safe/cache.rb:56:in `fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/memory.rb:60:in `fetch'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/memoizable-0.4.2/lib/memoizable/method_builder.rb:116:in `block (2 levels) in create_memoized_method'
     # /Users/tansaku/.rvm/gems/ruby-2.4.2/gems/twitter-text-2.0.2/lib/twitter-text/validation.rb:41:in `parse_tweet'
     # ./app/services/twitter_api.rb:22:in `post_to_twitter'
     # ./spec/services/twitter_api_spec.rb:17:in `block (3 levels) in <top (required)>'

It seems like when we have the twitter gem in a rails app that this somehow interferes with the twitter-text gem - so that configuration gets looked up in the twitter gem and that breaks for some reason. When we remove the twitter gem from the rails app then we don't get the above error.

-Adding babel-polyfill for supporting older browsers -Resolves #247 -Resolves error ReferenceError: Can't find variable: Symbol in bower_components/twitter-text/js/pkg/twitter-text-2.0.2.js (line 854)

Just doing require 'twitter-text' fails because symbolic link destination oflib/assets/tld_lib.yml that is lib/assets/../../../conformance/tld_lib.yml does not exsit as follows.

/Users/nyoho/.rbenv/versions/2.3.3/lib/ruby/2.3.0/psych.rb:474:in `initialize': No such file or directory @ rb_sysopen - /Users/nyoho/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/twitter-text-1.14.3/lib/assets/tld_lib.yml (Errno::ENOENT)

It occurs from version 1.14.3. It might be better to replace the symlink with a real file. (?)

Email Address Internationalization (EAI) is not hyperlinked correctly.

Here is the link to the tweet: https://twitter.com/datamail_in/status/1055140179089022976?s=19

The email address with non-ASCII character in the above tweet is valid and expected to have a hyperlink when typed in twitter application.

A quick guide that explains the expected behavior of a correct linkification is here: https://uasg.tech/wp-content/uploads/2017/06/UASG010-Quick-Guide-to-Linkification.pdf

It only linkifies until it comes across "á" letter. There is no link below "á" character.

Internationalized Domain Names (IDNs) permit the global community to use a domain name in their native language or script. This is enabled by allowing domain names to have characters from different scripts, beyond the letters (a to z), digits (0 to 9) and hyphen (-), as encoded by the Unicode standard and as allowed by relevant IDN protocols.

Email Address Internationalization (EAI) is the protocol that allows email addresses with IDNs in the domain part and/or Unicode (non-ASCII) characters in the Local part of the Mailbox name. Please see the quick guide for EAI from here: https://uasg.tech/wp-content/uploads/2017/02/UASG014_20170206.pdf

Kindly please look into this issue.

Thank you in advance.

Take as an example "Hand from back with index finger pointing right" emoji 👉

• When you use getTweetLength it is counted as 1 character
• But when you add this emoji directly into twitter.com tweet editor it is counted as 2 characters

Same problem can be found with multiple other emojis. As an example: 👩‍💻👨‍💻 are counted as 5 characters each by Twitter.com and 3 characters each by getTweetLength

It can be easily reproduced here: https://npm.runkit.com/twitter-text

var twitterText = require("twitter-text")

var text = "👉"

var tweetLength = twitterText.getTweetLength(text)
var jsLength = text.length

var response = `getTweetLength: ${tweetLength}, jsLength ${jsLength}`

1️⃣This emoji has not been accurately parse on parseTweet version: 3.0.1

Expected behavior

should be parsed as invalid

Actual behavior

This snippet is officially invalid, but the result of parseTweet is determined to be valid.

Steps to reproduce the behavior

1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣1️⃣

Do a parseTweet on this snippet

Ok, this one is kind of big, so it needs some explanation.

My original intent was to kill off twttr.txt.addCharsToCharClass and use RegEx literals when possible to make the library smaller and reduce startup coasts. That's when I realized in ruby, java, and objc, hashtags are being matched by the unicode-aware char classes for letters \p{L] and markers \p{M} but JavaScript doesn't have that, so is using a subset of that with no support for characters outside the basic multilingual plane (astral panes) that use surrogate pairs in UTF16.

The solution

1. Find a suitable substitute for the complete set of unicode letters and markers. I got this from https://github.com/slevithan/xregexp/blob/master/src/addons/unicode-categories.js I'm not sure where we got the "unicodeLettersAndMarks" list previously, but it is missing everything from the astral planes.
2. Alter the regular expressions for hashtags to use matching groups and negative matching groups instead of char classes and negative char classes. This was probably the most difficult part. The trickiest thing was how to negate a group in JavaScript. My solution was to do a negative lookahead followed by a match on the next complete code point (surrogate pair or normal char) This happens to work because the group can only match a single char unit or two units if it is in the astral plane.
3. Measure the increase in library size and reduction in match performance. The gzipped size goes up a little (8%) and the speed of matching goes down a little (15%) but I think it is worth it to have full coverage of all unicode hashtags. A major optimization (2x) was bypassing astral plane lookups if a bmp lookahead did not match Example: (?=#{non_bmp_code_pairs})(?:#{unicodeAstralLetters}|#{unicodeAstralMarks})
4. Add unit tests I added a couple conformance tests for astral plane letters and marks

I am particularly interested in hearing from @edengol since he did this excellent PR: https://github.com/twitter/twitter-text/pull/7 Sad to see him leave the flock, though https://twitter.com/edeng/status/705092008071696384

jackson-databind dependency adds 11347 methods to bundle. http://www.methodscount.com/?lib=com.fasterxml.jackson.core%3Ajackson-databind%3A2.8.7

This is devastating for android apps which usually aim to be below 64k method limit. I have substituted it with Gson 2.8.2, which only adds around 1345 methods

http://www.methodscount.com/?lib=com.google.code.gson%3Agson%3A2.8.0

(this is for 2.8.0, this site doesn't show it for 2.8.2, but I have counted it locally with dexcount-gradle-plugin and it has 832 methods after being proguarded).

message = "Let's get humans living on #mars t.co";
twttr.txt.parseTweet(message);

Same results if I add a path or a scheme to the url ( t.co/poo or http://t.co/poo).

Yields this error every other time entered in the console.

Uncaught TypeError: Cannot read property 'toASCII' of undefined at Object.toAscii var punycodeEncodedLabel = punycode.toASCII(label);

Edit: Just realized that these are all the same issue. Adding a path (as @codemonkey3045 has suggested) or a scheme doesn't help. https://github.com/twitter/twitter-text/issues/200 https://github.com/twitter/twitter-text/issues/171 https://github.com/twitter/twitter-text/issues/153

Hiya,

Now that 280 characters has rolled out to all users, can we get an update that handles this (and the special rules for what characters count double)?

Thanks, G

Bumps json5 from 2.1.0 to 2.2.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Environment For web, specify your OS and browser version. For mobile, specify device, OS and version. For libraries, what version of build tools are you using?

Additional context Add any other context about the problem here.

Bumps qs from 6.5.2 to 6.5.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps decode-uri-component from 0.2.0 to 0.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.