1 ![afero logo-sm](https://cloud.githubusercontent.com/assets/173412/11490338/d50e16dc-97a5-11e5-8b12-019a300d0fcb.png)

     2 

     3 A FileSystem Abstraction System for Go

     4 

     5 [![Build Status](https://travis-ci.org/spf13/afero.svg)](https://travis-ci.org/spf13/afero) [![Build status](https://ci.appveyor.com/api/projects/status/github/spf13/afero?branch=master&svg=true)](https://ci.appveyor.com/project/spf13/afero) [![GoDoc](https://godoc.org/github.com/spf13/afero?status.svg)](https://godoc.org/github.com/spf13/afero) [![Join the chat at https://gitter.im/spf13/afero](https://badges.gitter.im/Dev%20Chat.svg)](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

     6 

     7 # Overview

     8 

     9 Afero is an filesystem framework providing a simple, uniform and universal API

    10 interacting with any filesystem, as an abstraction layer providing interfaces,

    11 types and methods. Afero has an exceptionally clean interface and simple design

    12 without needless constructors or initialization methods.

    13 

    14 Afero is also a library providing a base set of interoperable backend

    15 filesystems that make it easy to work with afero while retaining all the power

    16 and benefit of the os and ioutil packages.

    17 

    18 Afero provides significant improvements over using the os package alone, most

    19 notably the ability to create mock and testing filesystems without relying on the disk.

    20 

    21 It is suitable for use in a any situation where you would consider using the OS

    22 package as it provides an additional abstraction that makes it easy to use a

    23 memory backed file system during testing. It also adds support for the http

    24 filesystem for full interoperability.

    25 

    26 

    27 ## Afero Features

    28 

    29 * A single consistent API for accessing a variety of filesystems

    30 * Interoperation between a variety of file system types

    31 * A set of interfaces to encourage and enforce interoperability between backends

    32 * An atomic cross platform memory backed file system

    33 * Support for compositional (union) file systems by combining multiple file systems acting as one

    34 * Specialized backends which modify existing filesystems (Read Only, Regexp filtered)

    35 * A set of utility functions ported from io, ioutil & hugo to be afero aware

    36 

    37 

    38 # Using Afero

    39 

    40 Afero is easy to use and easier to adopt.

    41 

    42 A few different ways you could use Afero:

    43 

    44 * Use the interfaces alone to define you own file system.

    45 * Wrap for the OS packages.

    46 * Define different filesystems for different parts of your application.

    47 * Use Afero for mock filesystems while testing

    48 

    49 ## Step 1: Install Afero

    50 

    51 First use go get to install the latest version of the library.

    52 

    53     $ go get github.com/spf13/afero

    54 

    55 Next include Afero in your application.

    56 ```go

    57 import "github.com/spf13/afero"

    58 ```

    59 

    60 ## Step 2: Declare a backend

    61 

    62 First define a package variable and set it to a pointer to a filesystem.

    63 ```go

    64 var AppFs = afero.NewMemMapFs()

    65 

    66 or

    67 

    68 var AppFs = afero.NewOsFs()

    69 ```

    70 It is important to note that if you repeat the composite literal you

    71 will be using a completely new and isolated filesystem. In the case of

    72 OsFs it will still use the same underlying filesystem but will reduce

    73 the ability to drop in other filesystems as desired.

    74 

    75 ## Step 3: Use it like you would the OS package

    76 

    77 Throughout your application use any function and method like you normally

    78 would.

    79 

    80 So if my application before had:

    81 ```go

    82 os.Open('/tmp/foo')

    83 ```

    84 We would replace it with:

    85 ```go

    86 AppFs.Open('/tmp/foo')

    87 ```

    88 

    89 `AppFs` being the variable we defined above.

    90 

    91 

    92 ## List of all available functions

    93 

    94 File System Methods Available:

    95 ```go

    96 Chmod(name string, mode os.FileMode) : error

    97 Chtimes(name string, atime time.Time, mtime time.Time) : error

    98 Create(name string) : File, error

    99 Mkdir(name string, perm os.FileMode) : error

   100 MkdirAll(path string, perm os.FileMode) : error

   101 Name() : string

   102 Open(name string) : File, error

   103 OpenFile(name string, flag int, perm os.FileMode) : File, error

   104 Remove(name string) : error

   105 RemoveAll(path string) : error

   106 Rename(oldname, newname string) : error

   107 Stat(name string) : os.FileInfo, error

   108 ```

   109 File Interfaces and Methods Available:

   110 ```go

   111 io.Closer

   112 io.Reader

   113 io.ReaderAt

   114 io.Seeker

   115 io.Writer

   116 io.WriterAt

   117 

   118 Name() : string

   119 Readdir(count int) : []os.FileInfo, error

   120 Readdirnames(n int) : []string, error

   121 Stat() : os.FileInfo, error

   122 Sync() : error

   123 Truncate(size int64) : error

   124 WriteString(s string) : ret int, err error

   125 ```

   126 In some applications it may make sense to define a new package that

   127 simply exports the file system variable for easy access from anywhere.

   128 

   129 ## Using Afero's utility functions

   130 

   131 Afero provides a set of functions to make it easier to use the underlying file systems.

   132 These functions have been primarily ported from io & ioutil with some developed for Hugo.

   133 

   134 The afero utilities support all afero compatible backends.

   135 

   136 The list of utilities includes:

   137 

   138 ```go

   139 DirExists(path string) (bool, error)

   140 Exists(path string) (bool, error)

   141 FileContainsBytes(filename string, subslice []byte) (bool, error)

   142 GetTempDir(subPath string) string

   143 IsDir(path string) (bool, error)

   144 IsEmpty(path string) (bool, error)

   145 ReadDir(dirname string) ([]os.FileInfo, error)

   146 ReadFile(filename string) ([]byte, error)

   147 SafeWriteReader(path string, r io.Reader) (err error)

   148 TempDir(dir, prefix string) (name string, err error)

   149 TempFile(dir, prefix string) (f File, err error)

   150 Walk(root string, walkFn filepath.WalkFunc) error

   151 WriteFile(filename string, data []byte, perm os.FileMode) error

   152 WriteReader(path string, r io.Reader) (err error)

   153 ```

   154 For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)

   155 

   156 They are available under two different approaches to use. You can either call

   157 them directly where the first parameter of each function will be the file

   158 system, or you can declare a new `Afero`, a custom type used to bind these

   159 functions as methods to a given filesystem.

   160 

   161 ### Calling utilities directly

   162 

   163 ```go

   164 fs := new(afero.MemMapFs)

   165 f, err := afero.TempFile(fs,"", "ioutil-test")

   166 

   167 ```

   168 

   169 ### Calling via Afero

   170 

   171 ```go

   172 fs := afero.NewMemMapFs()

   173 afs := &afero.Afero{Fs: fs}

   174 f, err := afs.TempFile("", "ioutil-test")

   175 ```

   176 

   177 ## Using Afero for Testing

   178 

   179 There is a large benefit to using a mock filesystem for testing. It has a

   180 completely blank state every time it is initialized and can be easily

   181 reproducible regardless of OS. You could create files to your heart’s content

   182 and the file access would be fast while also saving you from all the annoying

   183 issues with deleting temporary files, Windows file locking, etc. The MemMapFs

   184 backend is perfect for testing.

   185 

   186 * Much faster than performing I/O operations on disk

   187 * Avoid security issues and permissions

   188 * Far more control. 'rm -rf /' with confidence

   189 * Test setup is far more easier to do

   190 * No test cleanup needed

   191 

   192 One way to accomplish this is to define a variable as mentioned above.

   193 In your application this will be set to afero.NewOsFs() during testing you

   194 can set it to afero.NewMemMapFs().

   195 

   196 It wouldn't be uncommon to have each test initialize a blank slate memory

   197 backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere

   198 appropriate in my application code. This approach ensures that Tests are order

   199 independent, with no test relying on the state left by an earlier test.

   200 

   201 Then in my tests I would initialize a new MemMapFs for each test:

   202 ```go

   203 func TestExist(t *testing.T) {

   204 	appFS := afero.NewMemMapFs()

   205 	// create test files and directories

   206 	appFS.MkdirAll("src/a", 0755)

   207 	afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)

   208 	afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)

   209 	name := "src/c"

   210 	_, err := appFS.Stat(name)

   211 	if os.IsNotExist(err) {

   212 		t.Errorf("file \"%s\" does not exist.\n", name)

   213 	}

   214 }

   215 ```

   216 

   217 # Available Backends

   218 

   219 ## Operating System Native

   220 

   221 ### OsFs

   222 

   223 The first is simply a wrapper around the native OS calls. This makes it

   224 very easy to use as all of the calls are the same as the existing OS

   225 calls. It also makes it trivial to have your code use the OS during

   226 operation and a mock filesystem during testing or as needed.

   227 

   228 ```go

   229 appfs := afero.NewOsFs()

   230 appfs.MkdirAll("src/a", 0755))

   231 ```

   232 

   233 ## Memory Backed Storage

   234 

   235 ### MemMapFs

   236 

   237 Afero also provides a fully atomic memory backed filesystem perfect for use in

   238 mocking and to speed up unnecessary disk io when persistence isn’t

   239 necessary. It is fully concurrent and will work within go routines

   240 safely.

   241 

   242 ```go

   243 mm := afero.NewMemMapFs()

   244 mm.MkdirAll("src/a", 0755))

   245 ```

   246 

   247 #### InMemoryFile

   248 

   249 As part of MemMapFs, Afero also provides an atomic, fully concurrent memory

   250 backed file implementation. This can be used in other memory backed file

   251 systems with ease. Plans are to add a radix tree memory stored file

   252 system using InMemoryFile.

   253 

   254 ## Network Interfaces

   255 

   256 ### SftpFs

   257 

   258 Afero has experimental support for secure file transfer protocol (sftp). Which can

   259 be used to perform file operations over a encrypted channel.

   260 

   261 ## Filtering Backends

   262 

   263 ### BasePathFs

   264 

   265 The BasePathFs restricts all operations to a given path within an Fs.

   266 The given file name to the operations on this Fs will be prepended with

   267 the base path before calling the source Fs.

   268 

   269 ```go

   270 bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")

   271 ```

   272 

   273 ### ReadOnlyFs

   274 

   275 A thin wrapper around the source Fs providing a read only view.

   276 

   277 ```go

   278 fs := afero.NewReadOnlyFs(afero.NewOsFs())

   279 _, err := fs.Create("/file.txt")

   280 // err = syscall.EPERM

   281 ```

   282 

   283 # RegexpFs

   284 

   285 A filtered view on file names, any file NOT matching

   286 the passed regexp will be treated as non-existing.

   287 Files not matching the regexp provided will not be created.

   288 Directories are not filtered.

   289 

   290 ```go

   291 fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))

   292 _, err := fs.Create("/file.html")

   293 // err = syscall.ENOENT

   294 ```

   295 

   296 ### HttpFs

   297 

   298 Afero provides an http compatible backend which can wrap any of the existing

   299 backends.

   300 

   301 The Http package requires a slightly specific version of Open which

   302 returns an http.File type.

   303 

   304 Afero provides an httpFs file system which satisfies this requirement.

   305 Any Afero FileSystem can be used as an httpFs.

   306 

   307 ```go

   308 httpFs := afero.NewHttpFs(<ExistingFS>)

   309 fileserver := http.FileServer(httpFs.Dir(<PATH>)))

   310 http.Handle("/", fileserver)

   311 ```

   312 

   313 ## Composite Backends

   314 

   315 Afero provides the ability have two filesystems (or more) act as a single

   316 file system.

   317 

   318 ### CacheOnReadFs

   319 

   320 The CacheOnReadFs will lazily make copies of any accessed files from the base

   321 layer into the overlay. Subsequent reads will be pulled from the overlay

   322 directly permitting the request is within the cache duration of when it was

   323 created in the overlay.

   324 

   325 If the base filesystem is writeable, any changes to files will be

   326 done first to the base, then to the overlay layer. Write calls to open file

   327 handles like `Write()` or `Truncate()` to the overlay first.

   328 

   329 To writing files to the overlay only, you can use the overlay Fs directly (not

   330 via the union Fs).

   331 

   332 Cache files in the layer for the given time.Duration, a cache duration of 0

   333 means "forever" meaning the file will not be re-requested from the base ever.

   334 

   335 A read-only base will make the overlay also read-only but still copy files

   336 from the base to the overlay when they're not present (or outdated) in the

   337 caching layer.

   338 

   339 ```go

   340 base := afero.NewOsFs()

   341 layer := afero.NewMemMapFs()

   342 ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)

   343 ```

   344 

   345 ### CopyOnWriteFs()

   346 

   347 The CopyOnWriteFs is a read only base file system with a potentially

   348 writeable layer on top.

   349 

   350 Read operations will first look in the overlay and if not found there, will

   351 serve the file from the base.

   352 

   353 Changes to the file system will only be made in the overlay.

   354 

   355 Any attempt to modify a file found only in the base will copy the file to the

   356 overlay layer before modification (including opening a file with a writable

   357 handle).

   358 

   359 Removing and Renaming files present only in the base layer is not currently

   360 permitted. If a file is present in the base layer and the overlay, only the

   361 overlay will be removed/renamed.

   362 

   363 ```go

   364 	base := afero.NewOsFs()

   365 	roBase := afero.NewReadOnlyFs(base)

   366 	ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())

   367 

   368 	fh, _ = ufs.Create("/home/test/file2.txt")

   369 	fh.WriteString("This is a test")

   370 	fh.Close()

   371 ```

   372 

   373 In this example all write operations will only occur in memory (MemMapFs)

   374 leaving the base filesystem (OsFs) untouched.

   375 

   376 

   377 ## Desired/possible backends

   378 

   379 The following is a short list of possible backends we hope someone will

   380 implement:

   381 

   382 * SSH

   383 * ZIP

   384 * TAR

   385 * S3

   386 

   387 # About the project

   388 

   389 ## What's in the name

   390 

   391 Afero comes from the latin roots Ad-Facere.

   392 

   393 **"Ad"** is a prefix meaning "to".

   394 

   395 **"Facere"** is a form of the root "faciō" making "make or do".

   396 

   397 The literal meaning of afero is "to make" or "to do" which seems very fitting

   398 for a library that allows one to make files and directories and do things with them.

   399 

   400 The English word that shares the same roots as Afero is "affair". Affair shares

   401 the same concept but as a noun it means "something that is made or done" or "an

   402 object of a particular type".

   403 

   404 It's also nice that unlike some of my other libraries (hugo, cobra, viper) it

   405 Googles very well.

   406 

   407 ## Release Notes

   408 

   409 * **0.10.0** 2015.12.10

   410   * Full compatibility with Windows

   411   * Introduction of afero utilities

   412   * Test suite rewritten to work cross platform

   413   * Normalize paths for MemMapFs

   414   * Adding Sync to the file interface

   415   * **Breaking Change** Walk and ReadDir have changed parameter order

   416   * Moving types used by MemMapFs to a subpackage

   417   * General bugfixes and improvements

   418 * **0.9.0** 2015.11.05

   419   * New Walk function similar to filepath.Walk

   420   * MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC

   421   * MemMapFs.Remove now really deletes the file

   422   * InMemoryFile.Readdir and Readdirnames work correctly

   423   * InMemoryFile functions lock it for concurrent access

   424   * Test suite improvements

   425 * **0.8.0** 2014.10.28

   426   * First public version

   427   * Interfaces feel ready for people to build using

   428   * Interfaces satisfy all known uses

   429   * MemMapFs passes the majority of the OS test suite

   430   * OsFs passes the majority of the OS test suite

   431 

   432 ## Contributing

   433 

   434 1. Fork it

   435 2. Create your feature branch (`git checkout -b my-new-feature`)

   436 3. Commit your changes (`git commit -am 'Add some feature'`)

   437 4. Push to the branch (`git push origin my-new-feature`)

   438 5. Create new Pull Request

   439 

   440 ## Contributors

   441 

   442 Names in no particular order:

   443 

   444 * [spf13](https://github.com/spf13)

   445 * [jaqx0r](https://github.com/jaqx0r)

   446 * [mbertschler](https://github.com/mbertschler)

   447 * [xor-gate](https://github.com/xor-gate)

   448 

   449 ## License

   450 

   451 Afero is released under the Apache 2.0 license. See

   452 [LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)