gofeed

gofeed

Gofeed: A Robust Feed Parser for Golang

gofeed is a powerful and flexible library designed for parsing RSS, Atom, and JSON feeds across various formats and versions. It effectively manages non-standard elements and known extensions, and demonstrates resilience against common feed issues.

Table of Contents

• Features
• Overview
• Basic Usage
• Advanced Usage
• Dependencies
• License
• Credits

Features

Comprehensive Feed Support

• RSS (0.90 to 2.0)
• Atom (0.3, 1.0)
• JSON (1.0, 1.1)

Handling Invalid Feeds

gofeed takes a best-effort approach to deal with broken or invalid XML feeds, capable of handling issues like:

• Unescaped markup
• Undeclared namespace prefixes
• Missing or illegal tags
• Incorrect date formats
• …and more.

Extension Support

gofeed treats elements outside the feed’s default namespace as extensions, storing them in tree-like structures under Feed.Extensions and Item.Extensions. This feature allows you to access custom extension elements easily.

Built-In Support for Popular Extensions For added convenience, gofeed includes native support for parsing certain well-known extensions into dedicated structs. Currently, it supports:

• Dublin Core: Accessible via Feed.DublinCoreExt and Item.DublinCoreExt
• Apple iTunes: Accessible via Feed.ITunesExt and Item.ITunesExt

Overview

In gofeed, you have two primary choices for feed parsing: a universal parser for handling multiple feed types seamlessly, and specialized parsers for more granular control over individual feed types.

Universal Feed Parser

The universal gofeed.Parser is designed to make it easy to work with various types of feeds—RSS, Atom, JSON—by converting them into a unified gofeed.Feed model. This is especially useful when you’re dealing with multiple feed formats and you want to treat them the same way.

The universal parser uses built-in translators like DefaultRSSTranslator, DefaultAtomTranslator, and DefaultJSONTranslator to convert between the specific feed types and the universal feed. Not happy with the defaults? Implement your own gofeed.Translator to tailor the translation process to your needs.

Specialized Feed Parsers: RSS, Atom, JSON

Alternatively, if your focus is on a single feed type, then using a specialized parser offers advantages in terms of performance and granularity. For example, if you’re interested solely in RSS feeds, you can use rss.Parser directly. These feed-specific parsers map fields to their corresponding models, ensuring names and structures that match the feed type exactly.

Basic Usage

Universal Feed Parser

Here’s how to parse feeds using gofeed.Parser:

From a URL

fp := gofeed.NewParser()
feed, _ := fp.ParseURL("http://feeds.twit.tv/twit.xml")
fmt.Println(feed.Title)

From a String

feedData := `<rss version="2.0">
<channel>
<title>Sample Feed</title>
</channel>
</rss>`
fp := gofeed.NewParser()
feed, _ := fp.ParseString(feedData)
fmt.Println(feed.Title)

From an io.Reader

file, _ := os.Open("/path/to/a/file.xml")
defer file.Close()
fp := gofeed.NewParser()
feed, _ := fp.Parse(file)
fmt.Println(feed.Title)

From a URL with a 60s Timeout

ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
fp := gofeed.NewParser()
feed, _ := fp.ParseURLWithContext("http://feeds.twit.tv/twit.xml", ctx)
fmt.Println(feed.Title)

From a URL with a Custom User-Agent

fp := gofeed.NewParser()
fp.UserAgent = "MyCustomAgent 1.0"
feed, _ := fp.ParseURL("http://feeds.twit.tv/twit.xml")
fmt.Println(feed.Title)

Feed Specific Parsers

If you have a usage scenario that requires a specialized parser:

RSS Feed

feedData := `<rss version="2.0">
<channel>
<webMaster>[email protected] (Example Name)</webMaster>
</channel>
</rss>`
fp := rss.Parser{}
rssFeed, _ := fp.Parse(strings.NewReader(feedData))
fmt.Println(rssFeed.WebMaster)

Atom Feed

feedData := `<feed xmlns="http://www.w3.org/2005/Atom">
<subtitle>Example Atom</subtitle>
</feed>`
fp := atom.Parser{}
atomFeed, _ := fp.Parse(strings.NewReader(feedData))
fmt.Println(atomFeed.Subtitle)

JSON Feed

feedData := `{"version":"1.0", "home_page_url": "https://daringfireball.net"}`
fp := json.Parser{}
jsonFeed, _ := fp.Parse(strings.NewReader(feedData))
fmt.Println(jsonFeed.HomePageURL)

Advanced Usage

With Basic Authentication

fp := gofeed.NewParser()
fp.AuthConfig = &gofeed.Auth{
  Username: "foo",
  Password: "bar",
}

Using Custom Translators for Advanced Parsing

If you need more control over how fields are parsed and prioritized, you can specify your own custom translator. Below is an example that shows how to create a custom translator to give the /rss/channel/itunes:author field higher precedence than the /rss/channel/managingEditor field in RSS feeds.

Step 1: Define Your Custom Translator

First, we’ll create a new type that embeds the default RSS translator provided by the library. We’ll override its Translate method to implement our custom logic.

type MyCustomTranslator struct {
  defaultTranslator *gofeed.DefaultRSSTranslator
}

func NewMyCustomTranslator() *MyCustomTranslator {
  t := &MyCustomTranslator{}
  t.defaultTranslator = &gofeed.DefaultRSSTranslator{}
  return t
}

func (ct *MyCustomTranslator) Translate(feed interface{}) (*gofeed.Feed, error) {
  rss, found := feed.(*rss.Feed)
  if !found {
    return nil, fmt.Errorf("Feed did not match expected type of *rss.Feed")
  }

  f, err := ct.defaultTranslator.Translate(rss)
  if err != nil {
    return nil, err
  }

  // Custom logic to prioritize iTunes Author over Managing Editor
  if rss.ITunesExt != nil && rss.ITunesExt.Author != "" {
    f.Author = rss.ITunesExt.Author
  } else {
    f.Author = rss.ManagingEditor
  }
  
  return f, nil
}

Step 2: Use Your Custom Translator

Once your custom translator is defined, you can tell gofeed.Parser to use it instead of the default one.

feedData := `<rss version="2.0">
<channel>
<managingEditor>Ender Wiggin</managingEditor>
<itunes:author>Valentine Wiggin</itunes:author>
</channel>
</rss>`

fp := gofeed.NewParser()
fp.RSSTranslator = NewMyCustomTranslator()
feed, _ := fp.ParseString(feedData)
fmt.Println(feed.Author) // Valentine Wiggin

Dependencies

• goxpp - XML pull parser
• x/net - charset conversion and HTML parsing
• testify - unit test enhancements

License

This project is licensed under the MIT License

Credits

• cristoper for his work on implementing xml:base relative URI handling.
• Mark Pilgrim and Kurt McKee for their work on the excellent Universal Feed Parser Python library. This library was the inspiration for the gofeed library.
• Dan MacTough for his work on node-feedparser. It provided inspiration for the set of fields that should be covered in the hybrid gofeed.Feed model.
• Matt Jibson for his date parsing function in the goread project.
• Jim Teeuwen for his method of representing arbitrary feed extensions in the go-pkg-rss library.
• Sudhanshu Raheja for supporting JSON Feed parser